AI Evaluation Analyst
VirtualVocationsSeattle, Washington, United States5 hours ago
Job type
- Full-time
Job description
A company is looking for an AI Agent Evaluation Analyst.
Key Responsibilities
Review evaluation tasks and scenarios for logic, completeness, and realism
Identify inconsistencies, missing assumptions, or unclear decision points
Help define clear expected behaviors (gold standards) for AI agents
Required Qualifications
Excellent analytical thinking regarding complex systems and logical implications
Familiarity with structured data formats, such as JSON / YAML
Experience with policy evaluation, logic puzzles, or structured scenario design
Background in consulting, academia, or research
Some understanding of scoring or evaluation in agent testing
Create a job alert for this search
Ai Analyst • Seattle, Washington, United States
Related jobs
We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the q...Show moreLast updated: 30+ days ago
Primary Skills : Deep Learning (Expert); Machine Learning (Expert), LLM(Expert), AI (Advance), Data Manipulation(Proficient), Model Development-(Proficient).
Duration : 12 Months with possible extensi...Show moreLast updated: 30+ days ago
USA Remote (Prefer PST Time Zone).This is a non-traditional Human Data Research Analyst role working with an exciting AI start-up based in San Francisco.
In this role your responsibilities will invo...Show moreLast updated: 13 days ago
Job Title : Data Analyst - IT / Data Analyst IV.Duration : 06 months of contract on W2.Data Quality Improvement : Define standards for data collection, consistency, and automation; improve accuracy and ...Show moreLast updated: 30+ days ago
Principal Tech BD For Applied AI Solutions Go-To-Market Team.This position is part of the AWS Specialist and Partner Organization (ASP).
Specialists own the end-to-end go-to-market strategy for thei...Show moreLast updated: 9 days ago
Press Tab to Move to Skip to Content Link.Select how often (in days) to receive an alert : .We are seeking a talented and detail-oriented Visual Data Evaluator to join our team.The ideal candidate wi...Show moreLast updated: 3 days ago USA Remote (Prefer PST Time Zone).This exciting new Lead role focuses on the managing of a 50-person team who are accountable for the training of an AI engine and coordinating work / hand-offs to alt...Show moreLast updated: 13 days ago We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the q...Show moreLast updated: 30+ days ago Primary Skills : Machine Learning (Expert), Generative AI( Proficient), Statistical Modeling, Python (Expert), SQL (Expert), (Intermediate), PyTorch ((Intermediate).
Duration : 9 months with possible ...Show moreLast updated: 30+ days ago Overview Join the Video Platform team to navigate the complexities of large scale distributed systems spanning multiple regions, managing vast data streams from billions of users.Work on data pipe...Show moreLast updated: 26 days ago USA Remote (Prefer PST Time Zone).The Technical Research Associate is a non-traditional role in which you will work on our AI data collection team.
You have a background in QA or data analysis and a...Show moreLast updated: 13 days ago 
OneForma is looking for participants in our paid projects in.To collect videos that will support the training and improvement of AI-powered technologies.
Competitive fixed payment per session (detai...Show moreLast updated: 30+ days ago
Eurofins Scientific is a network of independent companies providing a unique range of analytical testing.From the food you eat, to the water you drink, to the medicines you rely on, Eurofins labora...Show moreLast updated: 30+ days ago Be among the first 25 applicants.This range is provided by Mercor.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.
Mercor connects elite creative...Show moreLast updated: 3 days ago We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the q...Show moreLast updated: 30+ days ago Join a collaborative remote team working to make AI communication emotionally intelligent, culturally aware, and engaging.
Score AI messages for tone, empathy, and contextual awareness.Identify subt...Show moreLast updated: 4 days ago Primary Skills : Machine Learning (Expert), Generative AI( Proficient), Statistical Modeling, Python (Expert), SQL (Expert), (Intermediate), PyTorch ((Intermediate).
Duration : 9 months with possible ...Show moreLast updated: 30+ days ago 
Turn your passion for luxury into a career opportunity.Explore the world of premium brands and make a lasting impact in fashion, beauty, jewelry, or automobiles.
Join CXG, the global leader in custo...Show moreLast updated: 30+ days ago A career in IBM Software means you'll be part of a team that transforms our customers' challenges into industry-leading solutions.
Seeking new possibilities and always staying curious, we are a team...Show moreLast updated: 30+ days ago Our UX Research team is designing for the broad spectrum of human needs, which requires us to understand the behaviors of the people behind them.
Our researchers tackle some of the most complex chal...Show moreLast updated: 30+ days ago
- Promoted
Remote Commercial Banking Analyst - AI Trainer
Data AnnotationRenton, WashingtonRemote
Full-time +1
Applied Scientist IV : 25-04131 (No C2C)
Akraya IncBellevue, Washington, United StatesFull-time
Quick Apply
Human Data Research Analyst
Keywords StudiosSeattle, WA, USFull-time
Quick Apply
Data Analyst
Sunrise SystemsBellevue, Washington, United StatesFull-time
Quick Apply
- Promoted
Principal Field Enablement, GTM Acceleration, Applied AI Solutions
AmazonSeattle, WA, USFull-time
Visual Data Evaluator Job at Firstsource Solutions Ltd in Seattle
MediabistroSeattle, WA, United StatesFull-time
Project Lead - AI
Keywords StudiosSeattle, WA, USFull-time
Quick Apply
- Promoted
Remote Senior Financial Analyst - AI Trainer
Data AnnotationRenton, WashingtonRemote
Full-time +1
Applied Scientist : 25-04992
Akraya IncBellevue, Washington, United StatesFull-time
Quick Apply
Data Analyst, Video Platform - USDS Job at TikTok in Seattle
MediabistroSeattle, WA, United StatesFull-time
Technical Research Associate - AI
Keywords StudiosSeattle, WA, USFull-time
Quick Apply
- Promoted
User Research Studies in Washington
CentificSeattle, WA, USFull-time
Analyst I - Eurofins Environment Testing Northwest, Tacoma, WA
Eurofins Environment Testing Northwest, LLCUSA, Washington, TacomaFull-time
Mercor is hiring : Audio Evaluation Expert - Fully Remote in Seattle
MediabistroSeattle, WA, United StatesRemote
Full-time
- Promoted
Remote Financial Analyst - AI Trainer
Data AnnotationFederal Way, WashingtonRemote
Full-time +1
Hireio, Inc. is hiring : AI Communication Quality & Culture Evaluator in Seattle
MediabistroSeattle, WA, United StatesFull-time
Applied Scientist : 25-04992 (No C2C)
Akraya IncBellevue, Washington, United StatesFull-time
Quick Apply
Become a Luxury Brand Evaluator in Bellevue, WA- Apply Now
CXGBellevue, WA, USFull-time
Quick Apply
IBM Computing is hiring : Sr User Experience Researcher - AI in Bellevue
MediabistroBellevue, WA, United StatesFull-time
UX Researcher, Qualitative Job at META in Bellevue
MediabistroBellevue, WA, United StatesFull-time