Senior Machine Learning Engineer - IntelliJ AI

Key Responsibilities: Develop and implement evaluation frameworks for AI features, including metrics, experiments, and trace analysis. Identify and resolve model performance challenges, such as prompt drift and latency/quality trade-offs, transforming insights into practical enhancements. Conduct experiments with contexts and lightweight models to continuously refine our machine learning system. Serve as the ML liaison for product teams, adapting and scaling AI functionalities within JetBrains IDEs. Create and sustain small supporting models (e.g., re-rankers, classifiers) tailored for specific tasks. Collaborate with colleagues across ML, product, engineering, and analytics teams to implement improvements and assess their production impact. Stay informed on the latest research in LLMs, agents, and evaluation methodologies, integrating best practices into our processes. Mentor junior engineers and contribute to the cultivation of a positive team culture, enhancing our experimentation and evaluation practices. Qualifications: Minimum of 5 years experience as a Machine Learning Engineer, with a robust background in production-grade ML systems, particularly LLMs and agent architectures. Familiarity with LLM evaluation methodologies and frameworks. Ability to design and execute comprehensive experiments, including hypothesis formulation, data collection, analysis, and decision-making. Proficient in context-aware pipelines or conversational/agent systems. Strong programming skills in Python.