About the job
Please submit your CV in English and indicate your level of English proficiency.
At Mindrift, we specialize in connecting talented individuals with project-based AI opportunities in collaboration with leading technology firms. Our focus is on the rigorous testing, evaluation, and enhancement of AI systems. This role is project-based and does not offer permanent employment.
Opportunity Overview
As an Evaluation Scenario Writer, you will develop challenging coding test cases that rigorously evaluate AI coding systems:
- Assess and refine realistic coding tasks utilizing provided production codebases, ensuring they have practical scope, requirements, and information sources.
- Construct comprehensive functional tests that validate true end-to-end behavior and edge cases, going beyond superficial checks.
- Create “fair but challenging” scenarios where the AI has all necessary context but must navigate complex reasoning and information spread across multiple files and external resources.
- Analyze AI performance failures to discern areas of struggle versus mastery.
- Iterate based on feedback from expert QA reviewers who evaluate your contributions using seven quality criteria.
Desired Qualifications
This opportunity is ideal for seasoned developers, software engineers, or test automation specialists seeking part-time, non-permanent projects. Ideal candidates should possess:
- A degree in Computer Science, Software Engineering, or a related field.
- Over five years of experience in software development, with a strong emphasis on Python (pytest, async/await, subprocess, file operations).
- A solid background in Full-Stack development, balancing expertise in building React-based interfaces and robust back-end systems.
- Experience in writing tests (functional, integration) rather than merely executing them.
- Familiarity with Docker containers for local evaluations.
- Understanding of CI/CD processes, particularly GitHub Actions (triggers, labels, result interpretation).
- English proficiency at a B2 level.
How the Process Works
Apply → Pass qualifications → Join a project → Complete tasks → Get compensated.
Estimated Effort
Tasks for this project are estimated to require about 20 hours to complete, depending on their complexity. This estimate is flexible; you determine your work schedule. Tasks must be submitted by the deadline and satisfy the specified acceptance criteria to be approved.
Compensation
- Paid contributions, with rates up to $80/hour*.
- Compensation may be fixed or based on individual project needs.
- Some projects offer incentive payments.
*Note: Rates may vary depending on expertise, skills assessment, location, project requirements, and other factors. Highly specialized experts may receive higher rates, while lower rates might apply during onboarding or non-core project phases. Payment details will be specified per project.

