AI Evaluation Analyst

A company is looking for an AI Agent Evaluation Analyst.

Key Responsibilities

Review evaluation tasks and scenarios for logic, completeness, and realism

Identify inconsistencies, missing assumptions, or unclear decision points

Help define clear expected behaviors for AI agents and annotate cause-effect relationships

Required Qualifications

Excellent analytical thinking about complex systems and logical implications

Familiarity with structured data formats such as JSON / YAML

Experience with policy evaluation, logic puzzles, or structured scenario design

Background in consulting, academia, or research

Some understanding of scoring or evaluation in agent testing