Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.
Entry Level
Join Toloka AI as a Freelance Agent Evaluation Engineer! In this exciting remote position, you'll play a crucial role in enhancing AI systems by evaluating and providing feedback on various agents. Your insights will directly contribute to improving user experiences and the efficiency of our AI solutions.
Toloka AI is a forward-thinking company dedicated to improving artificial intelligence through human feedback. We believe in harnessing the power of crowdsourcing to refine AI systems and ensure they meet user needs effectively. Join us in making a significant impact in the AI landscape!
Browse all companies, explore by city & role, or SEO search pages. View directory listings: all jobs, search results, location & role pages.
Tailoring 0 resumes…
We'll move completed jobs to Ready to Apply automatically.
Search for AI Agent Testing Specialist
793 results
Cresta is dedicated to transforming customer interactions into strategic advantages by harnessing the full potential of contact centers. Our innovative platform merges advanced AI capabilities with human intelligence, allowing contact centers to uncover valuable customer insights and optimize their processes. We automate conversations and inefficient workflo…
Eziway
Eziway, based in Melbourne, develops software that simplifies salary packaging and payroll for organizations and their employees. As part of the Fluent Software Group within Valsoft Corporation, Eziway combines the resources of a global company with the agility of a focused product team. The company is expanding its use of artificial intelligence to improve both customer service and internal processes. This new AI Agent Builder role centers on designing and delivering AI agents that transform how Eziway operates. The position covers the entire lifecycle of these projects, from initial concept through deployment and ongoing refinement. The AI Agent Builder will work directly with senior leaders, including the Managing Partners at Fluent Software Group and Eziway’s CEO, and will have a clear voice in shaping the direction of AI initiatives. What you will do Design and launch AI agents for Eziway’s core business functions, starting with customer support and research and development, then expanding to other teams. Develop an intelligent customer support agent to handle and resolve inbound inquiries, aiming to improve response times and reduce manual workloads. Create R&D acceleration agents that help speed up research, synthesize information, and generate actionable insights. Build a scalable agent framework that can be adapted by different teams across the company. Lead the integration of voice and conversational AI technologies to enhance the experience for Eziway’s customers. Role highlights Greenfield position with the chance to deliver real-world AI systems, not just prototypes or models. Direct collaboration with senior leadership and influence over key technology decisions. Hands-on work shaping how AI supports both customers and internal teams at Eziway.
teamified
Join our dynamic team at Teamified as a Human Resources Specialist focused on testing and evaluation. This role offers an exciting opportunity to contribute to our innovative HR practices on Macquarie Island Station. You will play a crucial role in ensuring our HR processes are efficient and effective.Your responsibilities will include conducting thorough testing of HR systems and processes, identifying areas for improvement, and collaborating with management to implement solutions that enhance our workforce management.We are seeking a proactive individual who is passionate about HR and is dedicated to fostering a positive work environment. You will be responsible for troubleshooting issues, providing insightful feedback, and supporting our HR initiatives.
About UsAt Sierra, we are revolutionizing the way businesses engage with their customers by creating a platform that enhances human interactions through AI technology. With headquarters in San Francisco and offices rapidly expanding in cities like Atlanta, New York, London, Paris, Madrid, Munich, Singapore, Japan, and Sydney, we are committed to fostering a collaborative and innovative environment.We operate under core values that shape our culture: Trust, Customer Obsession, Craftsmanship, Intensity, and Family. These principles guide our actions and reinforce our commitment to excellence.Founded by industry leaders Bret Taylor and Clay Bavor, Sierra is backed by extensive experience in technology and product innovation. Bret has held key positions including co-CEO of Salesforce and CTO of Facebook, while Clay has a strong background at Google, leading projects like Google Lens and AR/VR initiatives.Your RoleDesign and Deliver Production-Grade AI Agents: You will create and deploy highly efficient, reliable, and user-friendly AI agents that are vital to Sierra’s growth. These systems are not prototypes; they are robust, scalable solutions utilized across various sectors including finance, healthcare, and commerce.Drive the Agent Development Life Cycle (ADLC): You will take full ownership of the development process, from initial pilot projects to deployment and ongoing refinement. Your role encompasses building, tuning, and enhancing AI agents in live environments, setting the benchmark for ADLC best practices.Collaborate with Enterprises and Startups: You will engage with leaders from major corporations to comprehend their significant business challenges and develop AI agents that revolutionize their operations. Additionally, you will partner with pioneering startups to embed innovative solutions.
Toloka AI seeks a Freelance Civil Engineer and Python Specialist for contract work on AI training projects. This remote position is open to candidates located in Queensland, Australia. The role brings together civil engineering experience and Python programming to support the development and improvement of AI training solutions. Responsibilities Use civil engineering knowledge to contribute to AI training initiatives Write and refine Python code to build, validate, or improve AI training data and workflows Work with Toloka AI teams to achieve reliable, high-quality project outcomes Location This is a remote contract role for candidates based in Queensland, Australia.
Please submit your CV in English and include your English proficiency level. Mindrift, part of Toloka AI, matches experienced professionals with project-based AI work for major technology clients. This opening is for a specific project, not a permanent staff role. Role overview This project needs a Senior Python Systems Developer skilled in functional testing. The work centers on building and maintaining black box tests for large codebases across multiple languages. Strong Linux and Docker abilities are essential, along with experience interpreting code in C, Rust, or Go using large language models (LLMs). The team uses tools such as Roo Code and Claude Code to support fast, iterative development and migration tasks. What you will do Create and run functional black box tests for diverse codebases. Set up and manage Docker environments to ensure builds are reproducible and testing works across platforms. Track code coverage and implement automated metrics to meet industry standards. Use LLMs (including Roo Code and Claude) to automate tasks, accelerate development, and improve code quality. Requirements At least 5 years of professional experience as a Software Engineer, with a strong focus on Python. Advanced skills with pytest, including fixtures, session-scoped tests, and timeouts, plus experience designing black-box functional tests for CLI tools. Expertise in Docker: writing reproducible Dockerfiles, managing user contexts, and building secure workspaces. Deep knowledge of Linux and Bash scripting, comfortable with debugging in containers. Familiarity with modern Python tools such as uv, pyproject.toml, and packaging workflows. Ability to read and understand C, C++, Rust, or Go code using LLMs. Hands-on experience with LLMs (Claude Code, Roo Code, or Cursor) to drive iterative development and generate test cases. English proficiency at B2 level or above. Preferred qualifications Experience with agent evaluation platforms and MCP CLI. Tools and technologies Python (pytest, uv, Pillow), Docker, Bash, Git Submodules, C/C++/Rust/Go (for code reading), Dagger, GitHub Codespaces, LLMs (Claude Code, Roo Code, Cursor), coverage.py, gcov, kcov. Project details and benefits Freelance, project-based engagement through the Mindrift platform. Fully remote with flexible scheduling: set your own hours (20-30 per week). Compensation depends on project and expertise. For this project, AI trainers can earn up to $45 per hour.
Heidi Health
About Heidi:At Heidi, we are revolutionizing the healthcare landscape by creating an AI Care Partner that aids clinicians at every stage, from documentation to patient care delivery.Our mission is to enhance the capacity of healthcare while ensuring that care remains profoundly human. In just 18 months, we have empowered clinicians by reclaiming over 18 million hours and facilitating more than 73 million patient visits. Currently, Heidi supports over two million patient visits weekly across 116 countries and in more than 110 languages.Founded by healthcare professionals, our team combines the talents of clinicians, engineers, designers, scientists, and mathematicians, all united by a common goal: to fortify the human connections that are vital in healthcare.With nearly $100 million in funding, Heidi is rapidly expanding throughout the USA, UK, Canada, and Europe, collaborating with major health systems such as NHS, Beth Israel Lahey Health, MaineGeneral, and Monash Health.We prioritize agility in our operations while relying on proven methodologies to shape the future of healthcare. Are you ready to take on this challenge?Your Role:We are seeking a qualified physician or clinician with expertise in AI and machine learning to join our Medical Knowledge (MK) team as a Medical AI Specialist. You will leverage your clinical skills alongside your technical knowledge in software, AI, and machine learning within our fast-paced, high-growth startup environment.Key Responsibilities:Infuse medical insights into our AI products by combining clinical expertise with deep technical knowledge.Collaborate with cross-functional teams to translate clinician needs into impactful, reliable AI features.What You Will Do:Ensure the clinical integrity of Heidi: advise clinicians, resolve edge cases, and validate outputs.Work alongside product, engineering, and design teams to create prompts, build agentic AI, establish model evaluation frameworks, and refine models.Develop advanced AI features within Heidi utilizing the latest tools and frameworks.Plan and collaborate on research initiatives that demonstrate our AI scribe's clinical accuracy and user impact.Provide guidance to leadership on clinical quality and product roadmap priorities.Mentor team members and contribute to building a knowledgeable community.
Toloka AI is looking for a Freelance AI Trainer with a strong background in biology and Python. This remote position is based in Queensland, Australia and centers on advancing AI in the biological sciences. Role overview This role involves developing and refining AI models by applying expertise in both biology and programming. The work supports new advances in how AI is used within biological research and applications. What you will do Collaborate with a diverse team to design and improve training materials Participate in model training sessions, using knowledge of biology and Python to guide AI development Work environment This freelance position offers the flexibility of remote work and the chance to contribute meaningfully to the AI community, particularly at the intersection of technology and biological science.
Toloka-AI seeks a Computer Science Specialist with advanced Python skills to contribute to AI projects for the Mindrift platform. This remote role is open to candidates based in Queensland, Australia. Key responsibilities Use Python to develop and enhance AI technologies for the Mindrift platform. Work closely with team members to strengthen platform features and performance. Support improvements to user experience through technical input and solutions. Location This position is fully remote and based in Queensland, Australia.
Are you a medical professional passionate about revolutionizing the future of artificial intelligence? As large-scale language models transition from simple chatbots to robust scientific tools, your expertise can play a pivotal role in this transformation. By providing high-quality training data, we aim to democratize access to world-class education, align with the latest research, and enhance clinical workflows for healthcare practitioners globally. Your contribution will be crucial in shaping the next generation of AI. We are in search of dedicated medicine specialists with a deep understanding of internal medicine, pharmacology, pathology, clinical diagnostics, medical ethics, human physiology, epidemiology, immunology, and evidence-based medicine. You will engage with advanced language models on complex topics such as differential diagnosis, drug interactions, treatment protocols, pathophysiological mechanisms, clinical trial design, public health interventions, and diagnostic imaging interpretation, meticulously documenting every error to refine model reasoning. On a typical day, you will interact with the AI model on various clinical scenarios and theoretical medical inquiries, ensuring factual accuracy and logical integrity, capturing reproducible error traces, and proposing enhancements to our prompt engineering and evaluation metrics. An ideal candidate will possess a medical degree (MD or DO) or a master's/PhD in a health sciences field. Clinical experience, peer-reviewed publications, and involvement in hospital-based practices or public health projects will be advantageous. Clear and reflective communication skills, including the ability to articulate your reasoning, are essential. Are you ready to leverage your medical knowledge to contribute to the AI knowledge base of tomorrow? Apply now to help educate the AI model that will impact global health outcomes. We offer a competitive pay range of $8 to $65 per hour, with the final rate determined based on your experience, expertise, and location. Please note that as a contractor, you will need to provide a secure computer and high-speed internet; company-sponsored benefits such as health insurance and PTO do not apply.
System Canada Technologies
Join System Canada Technologies as a Test Analyst / Test Engineer and leverage your skills in Python scripting to enhance our testing processes. In this full-time position, you will play a crucial role in ensuring the quality and reliability of our software products. Collaborating with cross-functional teams, you will develop and execute test plans, identify defects, and contribute to continuous improvement efforts.
MYOB Group Limited
Join MYOB Group Limited as a Principal AI Security Specialist, where you will lead initiatives to develop and enhance security protocols for AI technologies. Your expertise will play a crucial role in safeguarding our innovative solutions and ensuring the integrity of our systems. Collaborate with cross-functional teams to identify vulnerabilities, implement security measures, and stay ahead of emerging threats in the AI landscape.
Join Mindrift, a platform connecting specialists with project-based AI opportunities in leading tech enterprises. Our focus is on optimizing, testing, and enhancing AI systems. This role is project-based and does not constitute permanent employment.Position OverviewAs a Supply Chain and Procurement Specialist, you will leverage your extensive experience to create authentic disruption scenarios, outline anticipated outcomes, and devise effective mitigation strategies. You will also assess AI-generated insights for their accuracy, comprehensiveness, and relevance to business operations.Key ResponsibilitiesCraft realistic supply chain disruption scenarios (e.g., supplier delays, quantity fluctuations, logistics challenges, quality failures) based on real-world manufacturing and procurement experiences.Define expected outcomes and mitigation strategies tailored to each scenario.Evaluate AI-generated recommendations against practical business logic.Review outputs for accuracy, completeness, and relevance within various ERP systems (notably Microsoft Dynamics 365, Coupa, Jaggaer, Ariba (SAP)).Assist in structured data creation and validation adhering to established guidelines.QualificationsA minimum of 4 years of relevant experience in procurement, supply chain, or purchasing, preferably within a manufacturing setting.Deep understanding of procurement workflows including purchase orders, vendor management, inventory, and production planning.Hands-on experience with ERP systems such as SAP, Oracle, or Microsoft Dynamics 365.Demonstrated ability to conceptualize and analyze supply chain disruptions and mitigation strategies.Solid grasp of disruption types: delays, shortages, quality issues, and logistics hurdles.Familiarity with Incoterms and logistics management.Understanding of Bill of Materials (BOM) structures and production planning processes.Experience with supplier performance metrics (e.g., OTIF, lead times, quality scores).Strong analytical skills to assess AI outputs against real-world business logic.Experience in data validation and structured data tasks.Excellent written communication skills in English.Application ProcessApply → Meet qualifications → Participate in a project → Complete tasks → Receive compensation.Project ExpectationsFor the active phases of this project, tasks are estimated to require approximately 10-20 hours per week, contingent on project needs. This is an estimate and not a guaranteed workload during active phases.CompensationEarn up to $60 per hour based on project involvement and expertise.
We invite you to submit your CV in English and specify your English proficiency level.At Mindrift, we connect talented professionals with innovative, project-based AI opportunities from leading technology companies, emphasizing the testing, evaluation, and enhancement of AI systems. Please note that participation in our projects is on a temporary basis and not permanent employment.Opportunity OverviewEach project presents unique challenges, and as a contributor, you may be tasked with:Designing original computational engineering problems that replicate real-world engineering workflows;Formulating problems that necessitate Python programming for engineering calculations and simulations;Creating computationally intensive problems requiring numerical methods or iterative approaches;Developing challenges related to system design, optimization, and analysis;Grounding problems in actual research challenges or practical engineering applications;Validating solutions using Python and standard engineering libraries;Clearly documenting problem statements and providing verified correct solutions.Qualifications We SeekThis role is ideal for engineers with proficiency in Python who are open to part-time, non-permanent assignments. The ideal candidates will possess:A degree in Mechanical Engineering or a related field;Proficiency in Python for numerical validation, with familiarity in MATLAB, R, C, SQL, Numpy, Pandas, SciPy, or any equivalent programming language;A minimum of 2 years of applicable professional experience, including applied work, research, or instructional roles;A solid understanding of practical engineering constraints and approximations;Exceptional written English skills (C1+ proficiency);Professional certifications (e.g., CMME, SAS Certifications, CAP) and experience with international or applied projects will be advantageous.Process OverviewApply → Complete qualification assessments → Join a project → Perform tasks → Receive compensation.Project Time ExpectationsDuring active phases, the tasks for this project are estimated to require approximately 10–20 hours per week, depending on project needs. This is an estimate and does not guarantee a specific workload.Compensation DetailsContributors can earn up to $39 per hour, based on their level and the pace of their contributions. Please be aware that compensation may vary for different projects based on the specific scope, complexity, and expertise required.
Toloka AI
Join Toloka AI as a Freelance Agent Evaluation Engineer! In this exciting remote position, you'll play a crucial role in enhancing AI systems by evaluating and providing feedback on various agents. Your insights will directly contribute to improving user experiences and the efficiency of our AI solutions.
Toloka-AI is seeking a Strategy Consultant to focus on AI training and evaluation. This remote role is available to candidates located in Queensland, Australia. Role overview This position centers on developing strategies that support AI training programs and enhance evaluation processes. The Strategy Consultant will work with teams across Toloka-AI to shape projects that impact the future of artificial intelligence. What you will do Create and refine strategies for AI training initiatives. Improve methods for evaluating AI systems and outcomes. Collaborate with experienced colleagues throughout the company. Contribute to projects that help guide the direction of AI development. Location This is a fully remote position for candidates based in Queensland, Australia.
Please submit your CV in English and specify your English proficiency level. Mindrift offers project-based freelance roles for mathematicians and Python specialists. Work remotely on AI projects for leading technology firms, focusing on testing, evaluating, and improving AI systems. All engagements are project-based and not permanent positions. What you will do Create original computational mathematics problems that mirror real mathematical research workflows Design challenges requiring Python programming solutions, often using libraries like Numpy, SciPy, and Sympy Develop problems that are computationally intensive and may take days or weeks to solve Craft problems involving complex reasoning in areas such as number theory, combinatorics, graph theory, and numerical analysis Base problems on real-world research questions or practical mathematical applications Validate solutions using Python and standard mathematical libraries Document problem statements clearly and provide verified solutions Requirements Degree in Mathematics (Pure or Applied) or a related field Strong Python skills for numerical validation; experience with MATLAB, R, C, SQL, Numpy, Pandas, SciPy, or similar languages is also considered At least 2 years of professional experience in applied mathematics, research, or teaching Familiarity with numerical methods and symbolic computation Ability to design problems reflecting authentic mathematical research processes Knowledge of computational complexity theory Excellent written English skills (C1+ level) Project workflow Application Qualification Assessment Project Assignment Task Completion Compensation Time commitment During active projects, tasks typically require 10–20 hours per week, depending on project needs. This is an estimate and not a guaranteed workload. Compensation Earn up to $35 per hour, depending on expertise and contribution speed. Actual rates vary by project scope, complexity, and required skills. Other projects on the platform may offer different compensation levels.
toloka-ai seeks a Senior Consultant for a freelance AI project. This remote position is open to candidates based in Queensland, Australia. The role centers on supporting strategic initiatives and contributing insights that help shape the company’s AI solutions. Role overview This position calls for a consultant with experience at MBB (McKinsey, BCG, Bain) or other leading consulting firms. The focus is on advising AI projects and influencing their direction through strategic guidance. Key responsibilities Advise on major strategic decisions for AI initiatives Apply consulting expertise to guide project direction Offer actionable insights to develop AI offerings Requirements Background at MBB or another top-tier consulting firm Strong experience in strategic consulting Comfort working independently as a freelancer
Veeam Software
Join Veeam, the Data and AI Trust Company, as we lead the charge in helping organizations maximize the potential of their data and AI assets. Our mission is to ensure that your data and AI are thoroughly understood, secured, and resilient, paving the way for the safe scaling of AI technologies. As the foremost authority in data resilience and security posture management, we are strategically positioned at the intersection of identity, data, security, and AI risk management. With our headquarters in Seattle and a presence in over 30 countries, we proudly safeguard the operations of more than 550,000 customers worldwide. Embark on a journey with us, as we strive to make a significant impact for top-tier global brands.In light of our recent acquisition of Securiti AI—an industry leader in AI-enhanced data security posture management (DSPM)—we are on the lookout for skilled Sales Specialists to propel growth in this cutting-edge sector.In this pivotal role, you will concentrate solely on Securiti AI offerings, collaborating closely with Veeam account executives to cross-sell our solutions to existing customers, attract new clients, and nurture current Securiti AI accounts. You will share a designated territory with several account executives, enjoy robust earning potential, and receive dedicated support from a Securiti AI solution engineer for technical engagements. Furthermore, you will leverage Veeam’s comprehensive go-to-market resources to ensure the success of our customers.
Join our dynamic team as a Freelance AI Trainer specializing in Australian English! This role is perfect for language enthusiasts who possess a deep understanding of the Australian dialect and culture. You will contribute to developing AI language models, providing insights, and ensuring the accuracy and relevance of language processing systems. Your expertise will help shape the future of AI communication.
Sign in to browse more jobs
Create account — see all 793 results
Browse all companies, explore by city & role, or SEO search pages. View directory listings: all jobs, search results, or location & role pages.
