About the job
Please submit your CV in English and specify your level of English proficiency.
At Mindrift, we specialize in connecting talented professionals with project-based AI opportunities for premier technology companies. Our primary focus is on the testing, evaluation, and enhancement of AI systems. Note: Positions are project-based and do not constitute permanent employment.
Your Role
As an Evaluation Scenario Writer, you will design rigorous coding test cases that challenge AI coding systems to excel:
- Evaluate and enhance realistic coding tasks derived from actual production codebases, ensuring they align with practical scopes, requirements, and relevant information sources.
- Devise thorough functional tests to validate the AI's end-to-end performance and edge cases, moving beyond superficial assessments.
- Create “fair yet challenging” tasks that equip AI with essential context while requiring it to apply complex reasoning across varied information sources.
- Investigate AI shortcomings to differentiate between areas of struggle and mastery.
- Refine your work based on feedback from expert QA reviewers who assess your contributions against seven key quality metrics.
Ideal Candidate Profile
This role is well-suited for seasoned developers, software engineers, or test automation specialists interested in part-time, non-permanent opportunities. Preferred qualifications include:
- A degree in Computer Science, Software Engineering, or a related field.
- Over 5 years of software development experience, focusing on Python (including pytest, async/await, subprocess, and file operations).
- A strong foundation in Full-Stack development, balancing expertise in both React-based front-end interfaces and robust back-end systems.
- Proficiency in writing tests, including functional and integration tests (not merely executing them).
- Experience with Docker containers for local evaluations.
- Understanding of CI/CD processes, particularly with GitHub Actions (triggers, labels, result interpretation).
- English proficiency at a B2 level or higher.
How This Works
Application → Qualification Assessment → Project Assignment → Task Completion → Payment
Estimated Effort
Project tasks are estimated at 20 hours, contingent upon complexity. This is a guideline, allowing you flexibility in when and how to work. All tasks must be submitted by their deadlines and meet acceptance criteria to be approved.
Compensation
- Compensated contributions with rates up to $17/hour*.
- Fixed project rates or individual rates depending on the project specifics.
- Some projects may include incentive payments.
*Note: Compensation rates vary based on expertise, skill assessments, location, project requirements, and other factors. Higher rates may be available for highly specialized professionals. Lower rates may apply during onboarding or initial project phases. Payment details will be provided per project.

