Talent.com
Mindrift
Software Engineer - AI Agent Evaluation (Remote)Mindrift • New York, NY, US
Software Engineer - AI Agent Evaluation (Remote)

Software Engineer - AI Agent Evaluation (Remote)

Mindrift • New York, NY, US
3 days ago
Salary
$50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

Software Engineer - AI Agent Evaluation (Remote) • New York, NY, US

Similar jobs

Senior AI Engineer (LLMs, RAG, Agentic Systems) - Remote (US)

Confidential Recruiting PartnersNew York City, NY, United States
Remote
Full-time

Our client is seeking a Senior AI Engineer to help design, deploy, and scale production-grade AI systems.This is a senior, product-focused engineering role with direct influence over how AI feature... Show more

 • Promoted

Clojure AI / LLM Engineer - REMOTE

StyliticsNew York City, NY, United States
Remote
Full-time

We are looking for a versatile and experienced AI / LLM Data Engineer to join our team and help shape the future of how Stylitics leverages large language models.In this role, you'll combine your e... Show more

 • Promoted

AI Workflow Developer (Remote) - Media & Entertainment

Tandym GroupEnglewood Cliffs, NJ, United States
Remote
Full-time

AI Workflow Developer (Remote) - Media & EntertainmentGet AI-powered advice on this job and more exclusive features.A recognized media organization offers a remote opportunity for an innovative... Show more

 • Promoted

Staff Software Engineer - Remote

TradeJobsWorkForce10034 New York, NY, US
Remote
Full-time

Staff Software Engineer Remote Job Duties: • Implement and evolve a Data Lake storage system with low latency and high thr... Show more

 • Promoted

AI Engineer (Remote)

Conch Technologies IncNew York City, NY, United States
Remote
Full-time

W2 Role only, No visa sponsorship at this time Role:AI Engineer Remote New York, NY, 10017 Bachelor s degree in Engineering, Statistics, Mathematics, Computer Science, Data Science, Economics, or a... Show more

 • Promoted

Senior Software Engineer - AI-Enabled Platform (Remote)

Franklin FitchNew York City, NY, United States
Remote
Full-time

A leading technology solutions provider is seeking a Senior Software Engineer to enhance an AI-enabled platform.The role involves converting discussions into actionable solutions and collaborating ... Show more

 • Promoted

Cloud & AI Applications Engineer (100% remote)

DBI StaffingNew York City, NY, United States
Remote
Full-time

We are seeking a motivated, technically savvy, and business-oriented professional to serve as the initial point of technical contact for the firm's cloud-based and AI-powered business applications.... Show more

 • Promoted

Consumer Insights Analyst

Earn HausPort Monmouth, New Jersey, United States
Part-time

We're currently seeking motivated individuals to participate in online surveys and paid gaming opportunities for well-known national brands.If you're looking for a flexible way to earn extra cash f... Show more

 • Promoted

Platform Backend Engineer - AI Audio, Remote

Clutch CanadaNew York City, NY, United States
Remote
Full-time

A leading text-to-speech company is looking for a Backend Developer to design and maintain robust APIs.This role involves working in a dynamic, remote environment, requiring expertise in TypeScript... Show more

 • Promoted

Full Stack Engineer (AI) - Remote

Siza- Buso ConsultingNew York City, NY, United States
Remote
Full-time

We are seeking a proactive and results-driven Full-Stack Engineer (AI) to join the client's AI team.The ideal candidate is a solid, well-rounded Full Stack Engineer who has evolved from web develop... Show more

 • Promoted

AI Engineer - Remote

Info Resume EdgeNew York City, NY, United States
Remote
Full-time

Freelancer Prox is the first platform that is completely free for both recruiters and job seekers.By creating a profile, your visibility increases globally, enabling faster hiring for both remote a... Show more

 • Promoted

AI Solutions Engineer - Remote

Finn PartnersMontvale, NJ, United States
Remote
Full-time

The AI Solutions Engineer is the primary builder, architect, and technical steward of the organization's internal AI capabilities.This role is responsible for the design, development, and operation... Show more

 • Promoted

Senior Software Engineer AI?Driven Platform (Remote/Hybrid)

MiterNew York City, NY, United States
Remote
Full-time

A leading software company is seeking a Senior Software Engineer in New York to design and implement innovative products.You will be responsible for leading cross-functional collaboration, improvin... Show more

 • Promoted

Staff ML Engineer, Agentic Systems (Remote-Friendly)

AnthropicNew York City, NY, United States
Remote
Full-time

A pioneering AI company in Seattle is seeking a Staff Machine Learning Engineer to enhance the capabilities of their AI systems.The ideal candidate will have extensive experience in machine learnin... Show more

 • Promoted

Analytics Engineer (Remote)

Serigor Inc.New York City, NY, United States
Remote
Full-time

Job DescriptionJob DescriptionJob Title :Analytics Engineer (Remote)Location :DC or ChicagoDuration :Full TimeJob Description :Our client is seeking an Analytics Engineer to develop, administer, an... Show more

 • Promoted

Director, Distinguished AI Engineer (Remote-Eligible)

Capital oneEnglewood Cliffs, NJ, United States
Remote
Full-time +1

Director Distinguished AI Engineer (Remote-Eligible)At Capital One we are creating responsible and reliable AI systems changing banking for good.For years Capital One has been an industry leader in... Show more

 • Promoted

Remote Product Tester - Flexible Studies

BuzzTestersLong Branch, NJ
Remote
Full-time

Earn up to $400/week, depending on the number and type of studies you complete.Share honest feedback on products and services from independent brands.Assignments include short online surveys (10–20... Show more

 • Promoted

Senior AI Product Engineer -- Browser UX & LLMs (Remote) Job at The Browser Compa

The Browser CompanyNew York City, NY, United States
Remote
Full-time

A forward-thinking tech company is seeking a Product Engineer to shape innovative AI features that redefine browser engagement.The ideal candidate will have over 6 years of experience in software p... Show more

 • Promoted

Sr Machine Learning / AI Engineer - Remote

Nava Software Solutions LLCJersey City, NJ, United States
Remote
Full-time

NAVA Software solutions is looking for a Senior Machine Learning / AI EngineerDetails :Senior Machine Learning / AI EngineerLocation :RemoteDuration :1yearAbout the ProjectThe team has developed an... Show more

 • Promoted

Remote | AI Platform & Integrations Software Engineer — $95–$140/hour

24-MAGNew York, New York, United States
$95.00 hourly
Remote
Full-time
Quick Apply

We are sharing a specialised full-time consulting opportunity for software engineers experienced in SaaS platform development, backend and frontend engineering, API integrations, real-time dashboar... Show more