About the job
Application Deadline: We are actively conducting interviews and aim to fill this position promptly once we identify a suitable candidate.
ABOUT THE ROLE
We are in search of talented Full-Stack Software Engineers who are passionate about developing tools for advanced AGI safety research. Your role will involve building and maintaining evaluation libraries and tools to monitor and manage our LLM traffic.
KEY PROJECTS
Your primary responsibility will be to create tools for analyzing model evaluation results. Here are some features you might develop and implement within your first 6 months:
- LLM-driven search functionality to locate intriguing segments in evaluation transcripts.
- Comparative views that illustrate discrepancies in conversations and scores across two evaluation runs.
- Capability to view and analyze dialogues with coding agents (e.g., Cursor, Claude Code) alongside evaluation transcripts.
- Real-time streaming of results for ongoing evaluations.
- Collaborative editing of evaluation logs that automatically updates metrics and other derived data.
Consider this as crafting an 'IDE for evaluations'.
In addition to the main projects, you may also explore auxiliary initiatives such as:
- Automated evaluation pipelines designed to reduce the time from accessing a new model for pre-deployment testing to analyzing key results and disseminating them.
- LLM agents and MCP tools that automate internal software engineering and research tasks, with safeguards to prevent significant failures.
- Telemetry API and enhancements to our existing tools to monitor usage and improve reliability.
- Upstream enhancements to the Inspect framework and ecosystem, including support for evaluating contemporary agentic scaffolds.
