1 - 20 of 41,806 Jobs

Search for Research Crawling Engineer

41,806 results

Apply
Wynd Labs logo
Full-time|Remote|Remote

Wynd Labs creates infrastructure that enables organizations to access large-scale web data for advanced AI model training. The team operates Grass, a bandwidth-sharing network powering a distributed web crawler that gathers high-quality public data from around the globe. Wynd Labs also manages data pipelines that process and annotate billions of videos, tran…

Apr 20, 2026
Apply
Coupang logo
Full-time|On-site|z-Test & Templates Only

Join us as a Crawling Engineer!At Coupang, we are dedicated to delivering an extraordinary shopping experience for our customers. As a Crawling Engineer, you will play a crucial role in enhancing our data acquisition processes and ensuring that our systems are optimized for performance and reliability.Application Instructions:Please fill out the Internal Transfer Request Form and submit it using your Coupang email address.

Mar 13, 2026
Apply
Exa logo
Full-time|On-site|San Francisco, California

At Exa, we are on a mission to create a groundbreaking search engine from the ground up, designed to empower every AI application. Our team develops robust infrastructure to efficiently crawl the web, trains cutting-edge embedding models for indexing, and engineers high-performance vector databases in Rust for seamless searching. We also manage a state-of-the-art $5M H200 GPU cluster that powers tens of thousands of machines.As a Software Engineer specializing in Web Crawling, you will take on the exciting challenge of building a web crawler that operates at a scale comparable to Google, revolutionizing the way we access and utilize information online.

Jul 23, 2025
Apply
Woflow logo
Full-time|Remote|Remote

Woflow is an innovative technology startup dedicated to developing cutting-edge products and solutions that cater to the rapidly evolving on-demand economy. Our flagship offering is a comprehensive platform that empowers customers to seamlessly request and receive vital merchant data, including structured menu information, images, and store details, through a robust combination of web applications and public APIs. At the core of this platform, our advanced machine learning and artificial intelligence models work in conjunction with an automated distributed workforce management system. We proudly stand as the world's first Merchant Data Platform.Our clientele spans food delivery services, online ordering platforms, and eCommerce marketplaces, as we provide the essential data infrastructure that fuels their growth and scalability.

Mar 6, 2025
Apply
Cloudflare, Inc. logo
Full-time|Hybrid|Hybrid

Join Cloudflare as a Frontend Engineer specializing in AI Crawl Control. In this pivotal role, you will develop innovative frontend solutions to enhance web crawling capabilities. Collaborate with a dynamic team of engineers and data scientists to create and optimize user interfaces that empower our AI systems. Your contributions will play a key role in shaping the future of web performance and security.

Mar 4, 2026
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization specializing in applied AI research, dedicated to two interrelated goals.The primary objective is to comprehend and develop intelligence. This entails establishing the mathematical foundations of reasoning, learning, decision-making, understanding, and explanation; as well as creating software that embodies these principles.The secondary goal is to enhance society's capacity to tackle complex challenges. This involves broadening the scale and complexity of the problems we can address today and, more importantly, accelerating our future problem-solving capabilities.To realize these ambitions, we are constructing a new technological base inspired by human reasoning and fostering a collaborative organization that prioritizes human values.About the RoleAs a Research Engineer in Operations at Basis, you will be instrumental in developing internal tools, automation, and measurement systems that significantly enhance researcher productivity and operational efficiency. Your work will involve creating tools that eliminate hours of manual tasks, automating proposal and CRM workflows, and offering analytics to guide strategic decisions as Basis expands.We seek individuals who are technically adept and possess a deep understanding of research workflows, enabling them to pinpoint high-impact automation opportunities. An ideal candidate has experience in converting research concepts into reliable code, building automation systems powered by LLMs, and designing user-friendly tools for researchers. Your contributions will range from researcher productivity enhancements to GTM automation and analytics infrastructure.This position is vital to Basis's growth strategy, transforming operations from manual to automated processes that facilitate scalable growth with sustainable results.We are looking for innovative thinkers who are technically accomplished and eager to explore foundational concepts. Our research engineers strive to develop high-quality, robust tools, unafraid to experiment, learn from mistakes, and explore unconventional ideas to achieve their goals.Basis thrives on collaboration, both internally and with our partners; we seek individuals who are passionate about empowering others to achieve milestones they couldn't accomplish alone.

Nov 23, 2025
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization dedicated to applied AI research, driven by two interlinked objectives.The first goal is to comprehend and construct intelligence. This involves establishing the mathematical foundations of reasoning, learning, decision-making, understanding, and explanation, alongside developing software that embodies these principles.The second objective is to enhance society's capacity to tackle complex challenges. This entails broadening the range, complexity, and scope of problems we address today, while importantly expediting our future problem-solving capabilities.To realize these ambitions, we are establishing a novel technological foundation inspired by human reasoning, as well as a unique collaborative organization that prioritizes human values.About the RoleAs a Research Engineer on the Platform team at Basis, you will propel research methodologies forward and package them into modular components for reuse by others. Your responsibilities include developing core technology modules (such as ChiRho, Effectful, Weighted), enhancing research-backed features for commercial platforms, and ensuring that research advancements lead to tangible impacts—both commercially and socially.We are seeking individuals who can merge research excellence with engineering precision. The ideal candidate has a background in both published research and production code, possesses the ability to translate experimental techniques into solid implementations, and thoughtfully approaches software architecture to facilitate others in building upon your contributions. You will identify high-impact research challenges aligned with Basis's mission and guide them from conception to concrete execution.This position is distinct from Operations Research Engineers (who concentrate on internal tools) as it emphasizes advancing research methods for platform and commercial applications. You will collaborate on Core Tech Module Teams and contribute to research that shapes the technological bedrock of Basis.We value individuals who excel at technical tasks and are unafraid to delve into foundational concepts. Our research engineers strive for thorough, high-quality, robust science and engineering, while embracing experimentation, learning from mistakes, and exploring unconventional ideas to achieve their goals.Basis thrives on collaboration, both internally and with our external partners; we seek individuals who enjoy working collectively on challenges that exceed individual capabilities.

Nov 23, 2025
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization dedicated to applied AI research, driven by dual objectives that enhance each other.Our primary focus is to understand and construct intelligence. This encompasses establishing the fundamental mathematical principles of reasoning, learning, decision-making, comprehension, and explanation, alongside developing software that embodies these principles.Secondly, we aim to empower society to tackle complex challenges. This involves broadening the scale, complexity, and scope of problems we can currently address and, crucially, accelerating our future problem-solving capabilities.To realize these ambitions, we are establishing an innovative technological framework inspired by human reasoning and fostering a collaborative environment that prioritizes human values.About the RoleAs a Research Engineer, you will play a vital role in advancing Basis’ mission by converting research concepts into accurate, robust, and scalable high-quality code.We are looking for technically proficient individuals who are enthusiastic about delving deep into foundational concepts. Our research engineers are committed to conducting rigorous, high-quality science, unafraid to experiment, learn from mistakes, and explore innovative ideas.Basis thrives on collaboration, both within our team and with external partners. We seek individuals who enjoy tackling challenges greater than they can manage alone.Machine Learning Research Engineer Focus AreasThis position is aimed at experts in machine learning engineering. Key areas of focus include:Probabilistic programming and statistical inferenceDeep learningCausal inferenceProgram synthesis and analysisML Ops and systems engineeringThese areas will be explored in the context of developing reasoning systems. Research engineers will also engage with topics including programming language design and implementation, automatic differentiation, and SAT/SMT solvers.

Feb 5, 2025
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization focused on applied AI research, driven by dual objectives.Our first goal is to comprehend and construct intelligence. We endeavor to lay down the mathematical principles defining reasoning, learning, decision-making, understanding, and explanation, while also developing software that embodies these principles.The second aim is to enhance society's capability to tackle complex challenges. We strive to broaden the scale, complexity, and diversity of problems we can address today, and more critically, to hasten our capacity to resolve future challenges.To fulfill these ambitions, we are creating an innovative technological foundation inspired by human reasoning, alongside a collaborative organization that prioritizes human values.About the RoleAs a Data Engineer on the Platform team at Basis, you will be responsible for constructing reliable data pipelines featuring comprehensive provenance and quality controls. You will curate documented datasets for training and evaluation while ensuring that our data infrastructure scales effectively. Your work will encompass both platform-specific data needs and cross-project data coordination, minimizing redundancy and fostering shared datasets.We are in search of technically proficient individuals who regard data quality as paramount. The ideal candidate has experience with machine learning data pipelines, understands the complete lifecycle from raw data to model training and evaluation, and approaches data provenance, lineage tracking, and quality assurance with rigor. You will blend software engineering best practices with a profound understanding of data systems and machine learning requirements.This role operates across both the Platform and Research teams, working on infrastructure that supports our commercial offerings and internal research initiatives. You will play a vital role in scaling Basis's data operations to support medium-scale models, ensuring data governance as we cater to external clients, and building systems that researchers can rely on for reproducible experiments.We seek individuals who are committed to executing high-quality, robust data engineering while remaining open to iteration, learning from real-world usage, and exploring diverse approaches to achieve excellence.At Basis, collaboration is key—both internally and with our external partners. We value those who enjoy laying the groundwork for solving grand challenges that extend beyond individual efforts.

Nov 23, 2025
Apply
Attic Projects logo
Full-time|$80K/yr - $100K/yr|On-site|Irving, TX

Attic Projects seeks a Field Superintendent for Attic and Crawl Space Restoration in Irving, TX. This salaried role reports to the Operations Director and focuses on leading field crews, supporting project delivery, and upholding company standards on-site. Attic Projects is ranked #117 among the top 500 remodelers in the U.S. The company is owner-led and values teamwork, providing training and support to help employees grow professionally and personally. The team is expanding in San Diego, Orange County, and both North and South Seattle, with a commitment to delivering high-quality solutions for clients. Role overview The Field Superintendent translates operational goals into clear direction for field teams. This position is responsible for project supervision, quality control, team leadership, preparation and planning of tasks, resolving issues as they arise, ensuring strong customer service, and managing documentation and reporting requirements. Key responsibilities Facilitate initial and ongoing training for the field workforce. Act as crew safety manager, ensuring compliance with OSHA standards. Oversee and organize equipment, tools, materials, and PPE at the facility. Conduct site audits and perform quality control (QC) visits. What to expect The role offers opportunities for skill development and career advancement within a supportive team. Attic Projects emphasizes learning, growth, and making a meaningful impact through daily work.

Apr 27, 2026
Apply
Bravos Research logo
Full-time|Remote|Remote — New York, New York, United States

Bravos Research stands at the forefront of investment research and financial media, specializing in video-first content. Our mission is to empower investors with insightful, data-driven research to navigate the complexities of global markets.As the proud creator of the largest investment research channel on YouTube, we have amassed over 75 million views and a substantial subscriber base for our premium research services.We are on the lookout for a Senior Research Analyst who can merge institutional-level macroeconomic insights with captivating storytelling. In this role, you will transform intricate economic data into engaging narratives and manage the end-to-end research and scriptwriting process for our animated videos, focusing on US and global macro trends, liquidity, credit markets, commodities, and cryptocurrencies.Key Responsibilities:Produce high-conviction insights regarding global business cycles, equity markets, central bank policies, and asset allocation strategies.Craft compelling, thoroughly sourced scripts that simplify complex economic concepts for a broad audience without compromising depth.Collaborate with team members to establish the firm's investment strategy and overarching perspectives.

Apr 30, 2025
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a nonprofit organization dedicated to applied artificial intelligence research. Our mission is twofold: to understand and build intelligence and to advance society’s problem-solving capabilities. We strive to unravel the mathematical principles behind reasoning, learning, decision-making, understanding, and explanation, while also developing software that embodies these principles.Our commitment extends to enhancing our ability to tackle complex issues that are beyond today's capabilities and accelerating our potential to address future challenges. We are creating a technological framework inspired by human reasoning, alongside fostering a collaborative organization that prioritizes human values.About the RoleAs a Research Scientist, you will spearhead Basis’ initiatives to deepen our understanding of the conceptual, mathematical, and computational principles of intelligence. We seek individuals who excel technically and are passionate about exploring foundational concepts. Our research team values rigorous, high-quality scientific endeavors while encouraging experimentation and exploration of innovative ideas.Basis thrives on collaboration, both internally and with external partners. We are looking for team players who enjoy tackling significant problems that require collective effort.Research FocusDespite the growing acknowledgment that acquiring and understanding world models is crucial for intelligence, current AI systems face challenges in mirroring this human capability. Key uncertainties remain regarding the essence of a world model, methods for reliably detecting its presence in agents, and approaches to develop agents capable of learning these models effectively.Our research, particularly within the MARA project, seeks to establish new foundations and technologies for modeling, abstraction, and reasoning in AI systems. MARA's goal is to identify principled methods for how intelligence constructs, refines, and employs world models through interactive experimentation. Achieving this will require advancements in knowledge representation, abstraction, reasoning, active learning, and reinforcement learning, necessitating a first-principles reevaluation of world modeling.

Oct 31, 2025
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a nonprofit organization dedicated to applied AI research, striving to achieve two interconnected objectives.Firstly, we aim to understand and develop intelligence. This involves establishing the mathematical foundations of reasoning, learning, decision-making, comprehension, and explanation; along with creating software that embodies these principles.Secondly, we seek to enhance society's capacity to address complex challenges. This means broadening the range, scale, and intricacies of the problems we can tackle today, while also accelerating our ability to solve future challenges.To fulfill these missions, we are constructing a novel technological framework inspired by human reasoning, and fostering a collaborative organization that prioritizes human values.About the RoleAs a Research Scientist, you will spearhead Basis's initiatives to deepen our comprehension of the theoretical, mathematical, and computational aspects of intelligence.We seek individuals with exceptional technical skills who are passionate about exploring concepts at their core. Our research scientists and engineers are committed to conducting rigorous, high-quality, and robust scientific inquiry, yet they embrace experimentation, learning from mistakes, and exploring innovative ideas to achieve their goals.Basis thrives on collaboration, both internally and with external partners; thus, we value team players who relish tackling challenges that surpass individual capabilities.Research FocusOur research under the MARA project is dedicated to forging new principles and technologies for modeling, abstraction, and reasoning in AI systems. The primary aim of MARA is to reveal principled methodologies for how intelligence constructs, refines, and applies world models through interactive experimentation.For this position, we specifically seek experts in Reinforcement Learning & Planning who can push the boundaries of model-based RL, exploration strategies, optimal control, and Bayesian optimization. You will develop agents capable of learning efficient policies in intricate, partially observable environments by utilizing structured world models.

Nov 23, 2025
Apply
OpenAI logo
Full-time|On-site|San Francisco

Overview:Join the dynamic Data Acquisition team at OpenAI, part of our Foundations organization, where you will play a crucial role in the data collection processes that power our model training initiatives. Our team is at the forefront of managing web crawling and GPTBot services, collaborating closely with departments such as Data Processing, Architecture, and Scaling. We are seeking a talented Software Engineer who is passionate about data acquisition and eager to make a significant impact.Key Responsibilities:Lead and innovate engineering projects focused on data acquisition, including web crawling, data ingestion, and search functionalities.Collaborate effectively with cross-functional teams, including Data Processing, Architecture, and Scaling, to ensure seamless data flow and operational efficiency.Partner with the legal team to navigate compliance and data privacy challenges.Design and implement highly scalable distributed systems capable of processing petabytes of data.Architect algorithms for efficient data indexing and robust search capabilities.Build and manage backend services for data storage, including working with key-value databases and ensuring synchronization.Implement solutions within a Kubernetes Infrastructure-as-Code environment and conduct regular system health checks.Conduct experiments and analyze data to derive insights that enhance system performance.Qualifications:Bachelor's, Master's, or PhD in Computer Science or a related field.4+ years of professional software development experience.Familiarity with large-scale web crawlers is a plus.Deep understanding of large stateful distributed systems and data processing methodologies.Proficient in Kubernetes and knowledgeable about Infrastructure-as-Code principles.Eager to explore and implement new technologies and approaches.Proven ability to manage multiple tasks and adapt to shifting priorities.Excellent written and verbal communication skills.About OpenAI:At OpenAI, we are pioneers in AI research and deployment, dedicated to ensuring that the advancements in artificial intelligence benefit humanity as a whole. Our mission is to push the boundaries of AI capabilities while adhering to safe and responsible deployment practices. Join us in our commitment to harnessing the power of AI for positive global impact.

Sep 22, 2023
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization dedicated to applied AI research, driven by a dual mission.Firstly, we aim to deepen our understanding and development of intelligence. This involves establishing the mathematical foundations of reasoning, learning, decision-making, understanding, and explanation, along with creating software that embodies these principles.Secondly, we strive to enhance society’s capacity to tackle complex challenges. This means broadening the scale and complexity of the problems we can address today, while also accelerating our future problem-solving capabilities.To realize these aims, we are constructing an innovative technological framework inspired by human reasoning and fostering a collaborative organization that prioritizes human values.About the RoleAs a Research Scientist at Basis, you will play a crucial role in advancing our understanding of the theoretical, mathematical, and computational principles underlying intelligence.Our Research Scientists possess key characteristics:Outstanding technical expertise—strong mathematical and computational foundations.A creative builder’s mindset—capable of designing, constructing, and refining complex systems based on foundational principles.Commitment to scientific rigor—engaging in high-quality, robust scientific inquiry without hesitance to experiment, learn from mistakes, and explore unconventional ideas.At Basis, collaboration is key, both within our teams and with external partners. We seek individuals who thrive in collaborative environments and are eager to tackle challenges that transcend individual capabilities.Programming Languages Research ScientistsThis role is tailored for experts in the design, implementation, and analysis of programming languages. You will assist in the design and implementation of the foundational computational reasoning systems under development at Basis.Key focus areas of programming languages research include compiler design, partial evaluation, program analysis, abstract interpretation, and program transformation. This work is conducted within the context of constructing reasoning systems, leading to engagement with topics such as probabilistic programming, automatic differentiation, and SAT/SMT solvers.Expectations:Possess a PhD (or equivalent experience) in a relevant field.

Jan 30, 2025
Apply
Anthropic logo
Full-time|On-site|San Francisco, CA

Join Anthropic as a Research Engineer focusing on Economic Research. In this role, you will leverage your analytical skills to conduct in-depth economic analysis and contribute to innovative projects aimed at enhancing our understanding of economic models and their implications.

Mar 12, 2026
Apply
Basis logo
Full-time|On-site|New York Office

About BasisBasis is a pioneering nonprofit organization dedicated to applied AI research, pursuing two interconnected goals.Our primary objective is to understand and create intelligence. This involves formulating the mathematical foundations of reasoning, learning, decision-making, understanding, and explanation, while developing software that embodies these concepts.The second goal is to enhance society’s capacity to solve complex challenges. We aim to broaden the scope, intricacy, and variety of challenges we can address today, and crucially, to expedite our ability to tackle future problems.To realize these objectives, we are constructing an innovative technological base inspired by human reasoning, and fostering a collaborative organizational culture that prioritizes human values.About the RoleAs a Research Scientist on the MARA (Modeling, Abstraction, and Reasoning Agents) initiative, you will be instrumental in developing computational theories of scientific reasoning in the realms of robotics and embodied intelligence. Your work will push the boundaries of world modeling, reinforcement learning, program synthesis, and robotic control, enabling systems to learn, reason about, and engage with the physical world.We are seeking outstanding researchers with a strong background in Program Synthesis & Neuro-symbolic Methods. The ideal applicant will have a robust publication record in esteemed venues, blending theoretical expertise with practical implementation skills, and a passion for building systems that emulate scientific learning—hypothesizing, experimenting, and modeling the workings of the world.In this role, you will collaborate with an interdisciplinary team to address fundamental inquiries: How can agents learn causal models through interaction? How do we connect high-level reasoning with low-level control? How can we produce interpretable, verifiable control programs instead of opaque policies?At Basis, we value collaboration, both within our team and with external partners; we seek individuals who thrive in teamwork on significant challenges that require collective effort.

Nov 23, 2025
Apply
OpenAI logo
Full-time|Hybrid|San Francisco

About the TeamJoin the innovative Post-Training team at OpenAI, where we focus on refining and elevating pre-trained models for deployment in ChatGPT, our API, and future products. Collaborating closely with various research and product teams, we conduct crucial research that prepares our models for real-world deployment to millions of users, ensuring they are safe, efficient, and reliable.About the RoleAs a Research Engineer / Scientist, you will spearhead the research and development of enhancements to our models. Our work intersects reinforcement learning and product development, aiming to create cutting-edge solutions.We seek passionate individuals with robust machine learning engineering skills and research experience, particularly with innovative and powerful models. The ideal candidate will be driven by a commitment to product-oriented research.This position is located in San Francisco, CA, and follows a hybrid work model requiring three days in the office each week. Relocation assistance is available for new employees.In this role, you will:Lead and execute a research agenda aimed at enhancing model capabilities and performance.Work collaboratively with research and product teams to empower customers to optimize their models.Develop robust evaluation frameworks to monitor and assess modeling advancements.Design, implement, test, and debug code across our research stack.You may excel in this role if you:Possess a deep understanding of machine learning and its applications.Have experience with relevant models and methodologies for evaluating model improvements.Are adept at navigating large ML codebases for debugging purposes.Thrive in a fast-paced and technically intricate environment.About OpenAIOpenAI is a pioneering AI research and deployment organization dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We are committed to pushing the boundaries of AI capabilities while prioritizing safety and human-centric values in our products. Our mission is to embrace diverse perspectives, voices, and experiences that represent the full spectrum of humanity, as we strive for a future where AI is a powerful ally for everyone.

Dec 1, 2025
Apply
OpenAI logo
Full-time|Hybrid|San Francisco

About Our TeamJoin the forefront of AI innovation with the RL and Reasoning team at OpenAI. Our team is dedicated to advancing reinforcement learning research and has pioneered transformative projects, including o1 and o3. We are committed to pushing the limits of generative models while ensuring their scalable deployment.About the RoleAs a Research Engineer/Research Scientist at OpenAI, you will play a pivotal role in enhancing AI alignment and capabilities through state-of-the-art reinforcement learning techniques. Your contributions will be essential in training intelligent, aligned, and versatile agents that power various AI models.We seek individuals with a solid foundation in reinforcement learning research, agile coding skills, and a passion for rapid iteration.This position is located in San Francisco, CA, and follows a hybrid work model of three days in the office per week. We also provide relocation assistance for new hires.You may excel in this role if:You are enthusiastic about being at the cutting edge of RL and language model research.You take initiative, owning ideas and driving them to fruition.You value principled methodologies, conducting simple experiments in controlled environments to draw trustworthy conclusions.You thrive in a fast-paced, complex technical environment where rapid iteration is essential.You are adept at navigating extensive ML codebases to troubleshoot and enhance them.You possess a profound understanding of machine learning and its applications.About OpenAIOpenAI is a pioneering AI research and deployment organization committed to ensuring that general-purpose artificial intelligence serves the greater good for humanity. We strive to push the boundaries of AI system capabilities while prioritizing safe deployment through our innovative products. We recognize AI as a powerful tool that must be developed with safety and human-centric principles, embracing diverse perspectives to reflect the full spectrum of humanity.We are proud to be an equal opportunity employer, welcoming applicants from all backgrounds without discrimination based on race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or any other legally protected characteristic.

May 14, 2025
Apply
Anthropic logo
Full-time|Hybrid|New York City, NY; New York City, NY | Seattle, WA; San Francisco, CA

Join our dynamic team at Anthropic as a Research Engineer or Research Scientist specializing in Tokens. In this role, you will engage in groundbreaking research and development, contributing to the advancement of our token systems and enhancing their capabilities. Your work will be pivotal in shaping the future of our technology and driving innovation in our projects.

Mar 12, 2026

Sign in to browse more jobs

Create account — see all 41,806 results

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.