AI Agent Evaluation Analyst

A company is looking for an AI Agent Evaluation Analyst.

Key Responsibilities

Review evaluation tasks and scenarios for logic, completeness, and realism

Identify inconsistencies, missing assumptions, or unclear decision points

Help define clear expected behaviors for AI agents

Required Qualifications

Excellent analytical thinking with the ability to reason about complex systems

Familiarity with structured data formats like JSON / YAML

Experience with policy evaluation, logic puzzles, or structured scenario design

Background in consulting, academia, or research

Some understanding of scoring or evaluation in agent testing