Data Engineer - World Models at moonvalley-ai | Remote

Your MissionData at Scale: Take ownership of the robust pipelines and storage systems that empower petabyte-scale multimodal datasets for model training.Sustainable Platforms: Develop automated and efficient tooling and systems that facilitate processing at scale while managing numerous small heterogeneous datasets.Essential QualificationsData Engineering Ex…

Are you driven by a passion for artificial intelligence and creativity? At Moonvalley, we are eager to connect with talented individuals who resonate with our vision. If you don't find a position that feels just right at this moment, we invite you to express your interest! We'll retain your information and reach out when a fitting opportunity arises.About MoonvalleyMoonvalley is at the forefront of developing innovative generative AI models that create Superbowl-level commercials and award-winning cinematic experiences. Our flagship HD model, Marey, is crafted from exclusively licensed and proprietary data tailored for use in Hollywood and enterprise applications.Our team represents an extraordinary blend of talents from various industries. With elite AI scientists from renowned companies such as DeepMind, Google, Microsoft, Meta, and Snap, we bring decades of combined experience in machine learning and computational creativity. We have also established the first AI-enabled movie studio in Hollywood, featuring accomplished filmmakers and visionary creatives. Collaborating with top producers, actors, and filmmakers in Hollywood, as well as globally recognized creative brands, we've successfully raised over $100 million from esteemed investors including General Catalyst, Bessemer, Khosla Ventures, and YCombinator. The journey has just begun!Who You’ll Collaborate WithAt Moonvalley, you will work alongside AI researchers, engineers, product strategists, marketers, designers, and other innovators. Whether you're constructing AI models, propelling business growth, or crafting our brand identity, you will engage with intelligent, driven individuals who appreciate bold concepts and innovation.Our MissionWe are redefining the landscape of creativity by providing AI tools that empower artists, creators, and businesses. From generative AI to digital storytelling, our initiatives are boundary-pushing and globally influential.Our Work CultureInnovation is the heartbeat of our operations, whether in AI, marketing, product development, or business processes. We foster a culture of ownership, experimentation, and creative problem-solving in a dynamic, collaborative environment.What We OfferCompetitive salary and equity optionsComprehensive private health insurancePension contributions (available in the UK, Canada, and the US)Unlimited paid vacation policyFully distributed and asynchronous work culture

Join moonvalley-ai as a Technical Staff Member focusing on Infrastructure and World Models. In this role, you will play a crucial part in designing, building, and maintaining our innovative AI infrastructure. We are looking for an individual who is passionate about technology and eager to contribute to groundbreaking solutions in AI.Your responsibilities will include collaborating with cross-functional teams to develop scalable systems, optimize performance, and ensure the reliability of our infrastructure. This is an exciting opportunity to work remotely and be part of a forward-thinking organization dedicated to transforming the AI landscape.

At Runway, we're at the intersection of art and science, pioneering AI that simulates the world. Our mission is to develop world models that are on the cutting edge of artificial intelligence progress. We believe that language models alone cannot tackle the most complex challenges like robotics, healthcare, and scientific discovery. True advancements require models that learn from their environment, just as humans do. This learning process can be significantly accelerated through simulation, leading to faster innovations in storytelling and scientific discovery.Our team is composed of creative, open-minded, and ambitious individuals who share a common goal of making a substantial impact on the world. If you’re motivated to build groundbreaking solutions and join a team that thrives on innovation, we want to hear from you!About the RoleWe are seeking a dedicated Design Engineer to become a part of Runway Labs, our innovative research team focused on advancing AI video technology and General World Models. This position is embedded within a dynamic laboratory setting that promotes exploration, experimentation, and technical ingenuity.Ideal candidates will be self-motivated and have experience collaborating with engineering and research teams to quickly develop and deploy prototypes. You will experiment with new user interfaces and interaction models for AI-generated video and world models, transforming cutting-edge research into practical, user-friendly applications.This role combines design skills with technical knowledge and a passion for both areas. Working in a small, agile team, you will partner with researchers, engineers, and product designers to create conceptual prototypes, design valuable interactive features, and develop tools that facilitate new creative workflows.Your ResponsibilitiesDesign and implement innovative user experiences at the convergence of AI video technology and emerging generative media.Develop tools and interfaces that empower researchers and creatives to engage with AI systems in real-time.Investigate new application domains for AI video and general world models, encompassing simulation, interactive media, and research tools.Push the boundaries of front-end engineering to expand possibilities within research and experimental contexts.

About the Institute of Foundation ModelsWe are a premier research facility focused on the development, comprehension, application, and risk mitigation of foundation models. Our objective is to propel research, cultivate the upcoming generation of AI innovators, and contribute significantly to a knowledge-centric economy.As a vital member of our team, you will engage with state-of-the-art foundation model training alongside distinguished researchers, data scientists, and engineers, addressing critical and transformative challenges in AI development. Participate in creating revolutionary AI solutions poised to redefine entire sectors. Your strategic and innovative problem-solving abilities will play a crucial role in establishing MBZUAI as a global leader in high-performance computing for deep learning, fostering impactful discoveries that motivate the next wave of AI visionaries.The RoleWe are the AllWorld Team within the Institute of Foundation Model (IFM) at MBZUAI. Our team is at the forefront of developing the PAN (Physical, Agentic, and Networked) world models—next-generation foundation models designed to unlock machine intelligence beyond traditional linguistic capabilities.Our mission is to confront the fundamental issues of world modeling and to establish a new paradigm for next-generation machine reasoning. We are seeking enthusiastic individuals who align with our vision and are excited to explore the boundaries of AI with us.

AI Research Engineer, World ModelsPalo Alto, CA (On-site)About 1XAt 1X, we are pioneering the development of humanoid robots designed to collaborate with humans, addressing labor shortages and fostering abundance across various industries.The RoleAs an AI Research Engineer specializing in world models, you will create expansive multi-modal generative models that project future sensor inputs and robotic actions derived from historical data. These foundational models empower robots to comprehend and navigate complex real-world environments. Your responsibilities will span data engineering, model architecture, and deployment, with the goal of enhancing robot autonomy. This position merges innovative research with pragmatic product development, challenging the boundaries of robotic intelligence.

Role Overview Turn new research into production-scale machine learning systems that can be deployed and maintained. Design and build end-to-end ML models and data pipelines, ensuring they are reliable and maintainable. Refine and optimize models focused on analyzing images and videos. Manage every stage of the machine learning workflow: experimentation, training, evaluation, and deployment. Develop low-latency inference systems for real-time applications, along with dependable ML infrastructure. Prototype quickly using open-source models, adapting them to meet product needs. Run experiments, analyze outcomes, and iterate to improve performance. Work closely with researchers and teams across product, engineering, and design to deliver scalable ML solutions. Keep up with new developments in machine learning and apply relevant advances to improve moonvalley-ai products.

Metaculus stands at the forefront of public forecasting, boasting more than 2,800,000 forecasts spanning over 20,000 questions. We function as an essential epistemic infrastructure akin to Wikipedia and Our World In Data, dedicated to delivering reliable and precise information across a spectrum of topics, particularly within the realms of science and technology. Our mission is to empower individuals to model, comprehend, predict, and effectively navigate the world’s most significant and intricate challenges.As a Research Engineer with a specialization in Forecast Coherence & World Models, you will be a trailblazer in crafting technologies that enhance the transparency and accessibility of forecasts and their foundational world models. This role positions you at the dynamic intersection of software engineering and epistemics, where you will develop tools that assist domain experts and forecasters in creating more coherent predictions, thereby facilitating improved decision-making on multifaceted, interdisciplinary issues.Your responsibilities will involve a strong emphasis on independent experimentation and rapid prototyping. You will conduct experiments using large language models (LLMs), design and test innovative methodologies, and create minimum viable products (MVPs) to validate conceptual ideas. Additionally, you will devise methods to elicit, aggregate, and visualize the interconnections among forecasts, including causal relationships and necessary conditions, while incorporating AI to aid in constructing world models and identifying critical variables that could enhance forecast accuracy.

About the Institute of Foundation ModelsWe are a pioneering research laboratory dedicated to the development, understanding, utilization, and risk management of foundation models. Our mission is to propel research forward, cultivate the next generation of AI innovators, and make transformative contributions to a knowledge-driven economy. As a member of our elite team, you will have the chance to engage with the forefront of foundation model training, collaborating with top-tier researchers, data scientists, and engineers to address some of the most pressing and impactful challenges in AI development. Your involvement will contribute to the creation of revolutionary AI solutions that can redefine entire industries. Your strategic and innovative problem-solving skills will be vital in positioning MBZUAI as a global leader in high-performance computing for deep learning, fostering discoveries that will inspire future AI trailblazers. The RoleYou will be responsible for the comprehensive strategy, design, and execution of our general-purpose world modeling initiatives. Your role will involve translating cutting-edge research (such as the PAN framework) into robust, production-ready simulators, leading a multidisciplinary team of engineers and scientists, and ensuring alignment with IFM’s overarching mission. Key ResponsibilitiesTechnical Leadership & VisionArticulate and refine the architecture of the world model, leveraging the principles of PAN:Multimodal data ingestion, mixed continuous/discrete representations, hierarchical generative modeling with an enhanced LLM backbone and diffusion-based predictors, generative loss grounded in real observations, and simulation for RL-based agent training.Establish performance, safety, and evaluation benchmarks, driving ongoing enhancements.Project & Team ManagementSteer the planning, resource allocation, and timelines for world modeling projects. Supervise research engineers and scientists (including data curators, RL specialists, and simulator developers) to facilitate cohesive advancement.Cross-Functional CollaborationCollaborate with agent, reasoning, and deployment teams to incorporate world model outputs into downstream applications (robotics, multi-turn dialogue systems, and autonomous technologies). Engage with external partners (from both academia and industry) to integrate the latest advancements and tools.

At Rhoda AI, we are pioneering the development of a comprehensive platform for the next generation of humanoid robots. Our ambition encompasses everything from high-performance, software-defined hardware to the foundational models and video world models that govern their operations. Our robots are engineered as versatile generalists, adept at navigating intricate, real-world environments and addressing scenarios that are not encountered during training. We operate at the confluence of large-scale learning, robotics, and systems, with a research team that includes esteemed researchers from Stanford, Berkeley, Harvard, and other renowned institutions. Rather than merely enhancing a feature, we are constructing an entirely new computing platform dedicated to physical tasks. With over $400M raised, we are aggressively investing in research and development, hardware innovation, and scaling up manufacturing to realize this vision.Key ResponsibilitiesLead research initiatives focused on foundational models and world models for robotics, including representation learning, dynamics/prediction, planning, and control.Define research challenges and formulate hypotheses rooted in real-world robotic autonomy requirements.Design and execute rigorous experiments at scale, encompassing ablations, benchmarking, and evaluation methodologies.Develop and assess model architectures aimed at enhancing long-horizon predictions, rollout quality, and overall robotic task performance.Investigate and improve pre-training and post-training processes, including fine-tuning, alignment, and evaluation of large multimodal models.Collaborate closely with Research Engineers to translate innovative ideas into scalable training pipelines and dependable systems.Effectively communicate research findings through internal documentation, presentations, and reviews.Publish and present research at prestigious venues.Required QualificationsPh.D. in a relevant discipline such as Machine Learning, Robotics, Computer Science, Electrical Engineering, Applied Mathematics, or Computer Vision.Demonstrated strong publication record in high-quality research (e.g., NeurIPS, ICML, ICLR, CoRL, RSS, ICRA, CVPR).In-depth knowledge of current machine learning techniques, particularly in areas such as:Deep learning and representation learning.Sequence modeling and transformers.Generative modeling (e.g., diffusion, autoregressive, latent-variable models).

About General IntuitionGeneral Intuition is at the cutting edge of AI research, dedicated to developing foundational models that excel in deep spatial and temporal reasoning. Over the past year, we have advanced the capabilities of AI agents that navigate complex environments, created world models that serve as training grounds for these agents, and innovated video understanding models aimed at real-world application.We are proud to have raised a seed funding round of $133 million from General Catalyst and Khosla, driving our mission to uncover the next generation of intelligence.What We're Looking ForMinimum of 5 years of experience in deep learning research or reinforcement learning, specifically with embodied agents or simulation environments.Solid foundation in representation learning and generative modeling, particularly using architectures like diffusion models, VAEs, and transformers applied to video data.Experience with world models and predictive control — you possess knowledge on training models that simulate dynamics and plan actions within learned environments.Proficiency in reinforcement learning (RL, model-based RL, or imitation learning), coupled with the capability to design and evaluate policy networks.Excellent programming skills in Python and deep learning frameworks such as PyTorch.Strong experimental capabilities — adept in handling large-scale training, evaluation pipelines, and managing intricate datasets or simulations.Publications or contributions to open-source projects in domains such as world modeling, simulation learning, or agent policies are a significant advantage.In-person requirement: We are specifically seeking candidates located in New York City, with a commitment to working in the office five days a week.Ownership & scientific rigor: You are committed to seeing ideas through from conception to proof of concept to deployment. You write clean, reproducible code and uphold high standards for experimental validity.Performance and scaling mindset: You understand how research can be translated into production systems, with a keen awareness of compute efficiency, distributed training, and potential data bottlenecks.Curiosity-driven and results-oriented: You thrive on tackling open-ended problems, while also knowing how to set measurable goals and deliver impactful systems.Passion for gaming & simulation: A strong interest in interactive environments and physics-based simulations is highly desirable.

About the Institute of Foundation ModelsAt the Institute of Foundation Models (IFM), we are committed to pioneering research related to building, comprehending, and managing foundation models effectively. Our goal is to push the boundaries of research, cultivate the next generation of AI innovators, and contribute significantly to a knowledge-driven economy.Joining our dynamic team means collaborating with top-tier researchers, data scientists, and engineers on the forefront of foundation model training. You will engage in developing innovative AI solutions that have the potential to transform entire industries. Your strategic problem-solving skills will play a crucial role in positioning MBZUAI as a global leader in high-performance computing for deep learning, facilitating groundbreaking discoveries that inspire future AI pioneers.The RoleWe are the AllWorld Team within the Institute of Foundation Models (IFM) at MBZUAI. Our team is at the cutting edge of developing PAN (Physical, Agentic, and Networked) world models—the next-generation foundation models that aim to expand machine intelligence beyond language.Our mission encompasses addressing the fundamental challenges associated with world modeling and establishing a new paradigm for next-generation machine reasoning. We invite passionate individuals who align with our vision and are eager to redefine the landscape of AI.

Role OverviewJoin World Golf Tour as a Data Analyst within our dynamic Product team. In this pivotal position, you will serve as the guardian of our data, meticulously organizing insights and analyzing telemetry to facilitate strategic business decisions. Your primary focus will be on creating and maintaining dashboards and analytical reports, collaborating with various teams across the studio and closely with the Product team to deliver actionable insights that propel our business forward. This role underscores the importance of robust data stewardship, visualization, and statistical analysis.Key Responsibilities• Clean, validate, and prepare datasets for analysis, addressing issues related to missing, inconsistent, or novel data.• Conduct exploratory data analysis to discover trends, patterns, and anomalies that guide business decisions.• Develop and sustain dashboards, reports, and visualizations using tools such as Amplitude, Power BI, or Excel.• Convert analytical findings into clear, actionable insights for both technical and non-technical stakeholders.• Collaborate with business teams (e.g., marketing, product, finance) to understand data needs and deliver pertinent analyses.• Assist with ad hoc analyses and deep dives to answer specific business questions or identify opportunities.• Ensure adherence to data governance, privacy, and security protocols.Qualifications• Bachelor’s degree in Data Analytics, Statistics, Mathematics, Computer Science, Economics, or a related quantitative discipline.• 2–4 years of experience in a data analyst or similar role, ideally within the gaming or software development sectors.• Solid proficiency in SQL for effective data querying and manipulation.• Familiarity with data analysis tools or languages such as Python or R.• Advanced Excel skills, including pivot tables, formulas, and data modeling.• Experience with data visualization tools like Tableau or Power BI.• Strong command of statistical methodologies and data analysis techniques.• Exceptional problem-solving and critical thinking abilities.• Excellent communication skills, with the capacity to present complex data clearly and concisely.Preferred Qualifications• Experience with data warehousing concepts and tools (e.g., Snowflake, Redshift).

Waymo is at the forefront of autonomous driving technology, committed to becoming the world's most trusted driver. Originating from the Google Self-Driving Car Project in 2009, we have dedicated ourselves to developing the Waymo Driver—The World’s Most Experienced Driver™. Our mission is to enhance mobility access while significantly reducing traffic-related fatalities. With over ten million rider-only trips powered by the Waymo Driver, we have accumulated extensive experience, having driven more than 100 million miles on public roads and tens of billions in simulation across 15+ U.S. states.The Waymo AI Foundations team is dedicated to creating machine learning solutions that tackle complex challenges in autonomous driving. Our goal is to ensure the safe operation of Waymo vehicles in various cities and under diverse driving conditions. We actively collaborate with other research teams within Alphabet, focusing on areas such as reinforcement learning, learning from demonstration, generative modeling, Bayesian inference, hierarchical learning, and robust evaluation.This position follows a hybrid work schedule and reports directly to a Staff Research Scientist / Tech Lead Manager.

About Formation BioFormation Bio is a pioneering pharmaceutical company harnessing technology and AI to revolutionize drug development. As advancements in artificial intelligence and drug discovery yield more candidate drugs than the industry can feasibly progress through traditional clinical trials, Formation Bio recognizes that this bottleneck may hinder the availability of new medicines for patients. Established in 2016 as TrialSpark Inc., we have developed innovative technology platforms and processes designed to expedite every stage of drug development and clinical trials. By collaborating with pharmaceutical companies, research organizations, and biotechs, we in-license and develop drugs beyond clinical proof of concept, effectively bridging the gap between innovation and patient accessibility. Our endeavors are supported by prominent investors from both the pharma and tech industries, including a16z, Sequoia, Sanofi, Thrive Capital, Sam Altman, John Doerr, Spark Capital, SV Angel Growth, and others. Discover more about our vision and initiatives through the following resources:Our Vision for AI in PharmaOur Current Drug PortfolioOur Technology & PlatformAt Formation Bio, our core values drive our mission to transform the pharmaceutical landscape. Each team member contributes to our collective goal of accelerating the delivery of new treatments to patients. About the PositionWe are seeking a Senior Data Engineer to join our Scientific Data Intelligence (SDI) team at Formation Bio. In this role, you will play a critical part in converting Real World Data (RWD)—which includes electronic health records, claims, and other longitudinal patient data—into structured, analytics-ready datasets. You will collaborate closely with our Data Science team to model and transform data, conducting analyses that answer research questions, generate evidence, and support scientific decision-making across our drug portfolio.This position uniquely combines healthcare data engineering, real-world evidence analysis, and generative AI. A solid foundation in building reliable and scalable data pipelines is essential, alongside a hands-on approach to working directly with data—constructing cohorts and performing analyses to drive insights.

Join Field AI, where we are revolutionizing the future of robotics through intelligent AI systems. Our mission is to create transformative, risk-aware, and reliable robotics solutions that tackle complex challenges in real-world settings, enabling robots to unlock their full potential. We focus on innovative approaches that surpass conventional data-driven methodologies and purely transformer-based architectures. Our globally deployed solutions are already yielding remarkable results, continuously enhanced through practical field applications.Are you passionate about shaping the future of AI-powered robotics? Do you excel at the crossroads of advanced autonomy, real-world implementation, and AI-driven decision-making? We are looking for a Robotics AI Engineer specializing in Field Foundation Models and Dynamics Foundation Models. In this pivotal role, you will design, develop, and deploy state-of-the-art AI-driven planning and control frameworks, ensuring our robots navigate effectively in unpredictable and high-pressure environments. This is a unique opportunity to redefine embodied AI, collaborating with a talented team that values innovative thinking, practical execution, and an AI-focused engineering approach. If you’re excited about creating adaptive, intelligent autonomy that learns and evolves beyond traditional models, we want to connect with you!

Mission: We are in search of a talented and driven Senior Data Engineering Analyst to join our innovative team focused on large-scale AI model evaluation through our platform, Ground Truth Regression. The ideal candidate will possess a solid foundation in data engineering, machine learning principles, and statistical analysis. At Motional, the orchestration of extensive data analysis is crucial for advancing our machine learning-driven autonomous driving technology. Our robo-taxi ML models and software are developed by a dedicated team of engineers and deployed multiple times daily. The Ground Truth Regression team is responsible for providing a comprehensive perception analysis framework to validate any changes that could influence the behavior of autonomous vehicles. The Ground Truth Regression team assesses the cumulative effects of ML and software modifications during the development phase, equipping the team with tools for root cause analysis and safety evaluations. We meticulously monitor model errors, anomalies, and rare occurrences across extensive driving scenarios, gathering insights over thousands of driving hours. The team is responsible for developing the full-stack infrastructure, including AWS Kubernetes orchestration, re-simulation, model metrics, regression reports, and deep-dive analytical tools. Collaboration with other perception and ML teams is integral for establishing metrics and analyzing outcomes. What You’ll Do: Collaborate with ML Engineers and Autonomy Software Developers to create new data analysis metrics and KPIs for validating autonomous vehicle performance. Pursue innovation by researching and developing new large-scale data analysis frameworks. Lead large-scale data analysis workflows to identify unexpected impacts of both planned and unplanned modifications to ML models and software. Construct high-quality datasets to enhance ML product performance through training and edge case validation. Deliver in-depth statistical insights on model performance and generalization through thorough error and change analysis. What We’re Looking For: Bachelor's or Master’s degree in Computer Science, Data Science, or a related field. Proficient programming skills in Python, SciPy, and related data analysis frameworks. Experience in machine learning, data mining, and statistical analysis. Familiarity with large-scale data processing and distributed computing technologies. Strong communication and software development skills, including goal setting, design documentation, testing frameworks, and collaborative team development.

About BasisBasis is a nonprofit organization dedicated to applied artificial intelligence research. Our mission is twofold: to understand and build intelligence and to advance society’s problem-solving capabilities. We strive to unravel the mathematical principles behind reasoning, learning, decision-making, understanding, and explanation, while also developing software that embodies these principles.Our commitment extends to enhancing our ability to tackle complex issues that are beyond today's capabilities and accelerating our potential to address future challenges. We are creating a technological framework inspired by human reasoning, alongside fostering a collaborative organization that prioritizes human values.About the RoleAs a Research Scientist, you will spearhead Basis’ initiatives to deepen our understanding of the conceptual, mathematical, and computational principles of intelligence. We seek individuals who excel technically and are passionate about exploring foundational concepts. Our research team values rigorous, high-quality scientific endeavors while encouraging experimentation and exploration of innovative ideas.Basis thrives on collaboration, both internally and with external partners. We are looking for team players who enjoy tackling significant problems that require collective effort.Research FocusDespite the growing acknowledgment that acquiring and understanding world models is crucial for intelligence, current AI systems face challenges in mirroring this human capability. Key uncertainties remain regarding the essence of a world model, methods for reliably detecting its presence in agents, and approaches to develop agents capable of learning these models effectively.Our research, particularly within the MARA project, seeks to establish new foundations and technologies for modeling, abstraction, and reasoning in AI systems. MARA's goal is to identify principled methods for how intelligence constructs, refines, and employs world models through interactive experimentation. Achieving this will require advancements in knowledge representation, abstraction, reasoning, active learning, and reinforcement learning, necessitating a first-principles reevaluation of world modeling.

The Bot CompanyJoin us in creating a revolutionary robot designed to enhance everyday living.Based in San Francisco, our dynamic team consists of talented engineers, designers, and operators hailing from industry leaders like Tesla, Cruise, OpenAI, Google, and Pixar. We've successfully delivered exceptional products and experiences to hundreds of millions of users.Our deliberately streamlined team structure fosters prompt decision-making, eliminating bureaucracy and hierarchy. Each team member is an individual contributor empowered with significant scope, radical ownership, and direct accountability, working collaboratively across the stack in a fast-paced environment focused on rapid iteration and execution.What We Value in CandidatesAt The Bot Company, we seek individuals who exhibit remarkable sharpness and can thrive in high-pressure situations. Throughout the selection process, we expect candidates to showcase:Exceptional mental acuity: You think quickly, absorb information rapidly, and navigate unfamiliar domains with ease.Engineering curiosity: You possess an innate desire to understand system functionalities, extending beyond your primary expertise.High performance mindset: You excel in ambiguous environments and maintain high productivity under challenging conditions.Machine Learning Engineer: World ModelsWe are developing neural simulators capable of comprehending the fundamental

Role overview The Technical Staff Member (Data) - World Models position at Reka centers on building and maintaining large-scale data systems for AI model training. This role can be based in the US, Singapore, or remote. The focus is on integrating and processing petabyte-scale multimodal datasets, ensuring data infrastructure is both sustainable and efficient. What you will do Lead the development of data pipelines and storage solutions to handle vast and varied datasets for model training. Create automated, resource-efficient tools and systems that process diverse small datasets at scale. Design, automate, and maintain Python ETL pipelines (using Spark or Ray) for large multimodal data processing. Develop and maintain systems for data cataloging, lineage tracking, quality assurance, integrity checks, access management, and lifecycle oversight. Support colleagues by providing internal tools, documentation, and guidance on data best practices. Act as the primary steward of the organization’s datasets, ensuring their quality, accessibility, and overall health. Key challenges Build high-performance pipelines that process petabyte-scale datasets across thousands of CPUs and hundreds of GPUs. Adapt data formats, storage, and processing methods to keep pace with AI advancements while maintaining backward compatibility. Scale data infrastructure to support rapid organizational growth. Ensure the platform remains flexible for handling heterogeneous datasets and ad-hoc analytics needs. Requirements Expertise in data engineering, especially with Python ETL pipelines and familiarity with infrastructure, data formats, and large-scale storage systems. Experience in managing datasets, annotations, and data versioning for machine learning model training. Solid understanding of fundamental machine learning concepts to collaborate effectively with researchers and inform platform decisions. Ability to draft clear specifications for AI agents and maintain strong human oversight of AI-generated outputs. Demonstrated initiative, ownership, and effective communication in managing workload and priorities.