AI/LLM Test Engineer at innopeaktech | Palo Alto, CA

Full-time Position Requirements:Core Testing & Evaluation:Design and execute performance tests for LLMs across diverse product use cases (e.g., chatbots, content generation). Develop automated test frameworks to evaluate LLM outputs for accuracy, bias, safety, and coherence. Conduct end-to-end testing of integrated generative AI solutions, including APIs, data pipelines, and user interfaces. Optimization & Validation:Collaborate with ML engineers to validate fine-tuned models and optimize prompts for targeted scenarios. Analyze model failures, edge cases, and adversarial inputs to identify potential risks and areas for improvement. Benchmark LLM performance against industry standards and product-specific KPIs. Collaboration & Quality Assurance:Work alongside product, engineering, and research teams to define testing requirements and acceptance criteria. Document defects, performance metrics, and testing results to foster data-driven improvements. Promote AI ethics and safety through thorough testing of fairness, bias mitigation, and content moderation. Innovation & Tooling:Create scalable tools for synthetic test data generation, prompt variation testing, and automated evaluation workflows. Stay abreast of advancements in generative AI testing, including red-teaming techniques and evaluation frameworks (e.g., HELM, Dynabench). Propose innovative testing strategies to address emerging challenges (e.g., hallucinations, context drift).