LLM Evaluation Engineer

About the Role :

We are seeking a LLM Evaluation Engineer to join a forward-thinking team responsible for developing a sophisticated voice assistant platform. This isn’t your typical QA role – it’s a unique blend of technical engineering, machine learning evaluation, and data analysis. You’ll work closely with cutting-edge conversational AI technology, designing evaluation frameworks, building custom scripts, and creating data visualizations to assess platform performance.

Key Responsibilities :

• Design and implement evaluation strategies for voice and language models, including automated testing approaches.
• Analyze unstructured data from log store systems to identify performance gaps and optimize user experiences.
• Build and maintain custom Python scripts to streamline data processing and generate actionable insights.
• Develop visual reports to communicate findings and drive continuous improvement.
• Collaborate with cross-functional teams globally to identify and address pain points in conversational AI performance.
• Use prompt engineering techniques to refine LLM outputs and articulate system health.

Ideal Candidate :