AI Evaluation Analyst

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

Review evaluation tasks and scenarios for logic, completeness, and realism

Identify inconsistencies, missing assumptions, or unclear decision points

Help define clear expected behaviors (gold standards) for AI agents

Required Qualifications

Excellent analytical thinking with the ability to reason about complex systems

Familiarity with structured data formats, such as JSON / YAML

Experience with policy evaluation, logic puzzles, or structured scenario design

Background in consulting, academia, or research

Exposure to LLMs, prompt engineering, or AI-generated content