Tavily logoTavily logo

Software Engineer - Web Crawling

TavilyIsrael
On-site Full-time

Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.


Experience Level

Experience

Qualifications

Ideal candidates will possess a strong background in software engineering with a focus on web technologies and automation. Familiarity with AI and data acquisition systems is highly desirable. Experience with tools such as Playwright and Puppeteer will be advantageous. Candidates should have a passion for problem-solving and a commitment to delivering high-quality results.

About the job

About Tavily

Tavily is pioneering the search engine designed specifically for AI agents, envisioning a future where AI leads the way in the workplace. Our innovative Search API empowers AI agents with real-time access to the web, tailored for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG).

Backed by prominent investors, we cater to developers and enterprises globally. Our dynamic and ambitious team thrives on rapid shipping, continuous iteration, and a deep commitment to impactful outcomes.

The Role

  • Develop systems that facilitate AI's interaction with the live web.
  • Design and implement large-scale, intelligent web acquisition pipelines to make sense of an ever-evolving online landscape and convert it into structured, high-quality data.
  • Combine technical expertise in browsers, networks, and automation with practical engineering skills.
  • Work on adaptive systems that seamlessly navigate modern websites, comprehend changing web interactions and detection systems, and gather valuable information safely, efficiently, and at scale.
  • Your contributions will significantly influence Tavily's ability to provide AI agents with accurate, real-time knowledge, bridging the gap between the open web and intelligent reasoning.
  • This position is based in Israel and requires existing legal authorization to work in the country. Unfortunately, Tavily is unable to offer visa sponsorship for this role.

What You'll Do

  • Create distributed data acquisition systems that capture and structure the live web.
  • Explore and analyze browser internals, fingerprinting, and anti-automation systems to develop stealthy, adaptive orchestration layers.
  • Prototype and deploy intelligent automation frameworks using Playwright, Puppeteer, and low-level browser control protocols (CDP).
  • Conduct hands-on research into network flows, JavaScript protections, and emerging web standards impacting automation.
  • Collaborate with AI and infrastructure teams to integrate real-time web data into retrieval pipelines and LLM-powered agents.
  • Translate in-depth technical insights into production-ready components, balancing research flexibility with engineering precision.

About Tavily

At Tavily, we are at the forefront of transforming how AI agents access information on the web. Our cutting-edge technology and ambitious team are dedicated to shaping the future of AI in the workplace.

Similar jobs

Browse all companies, explore by city & role, or SEO search pages. View directory listings: all jobs, search results, location & role pages.

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.