ML / LLM Operations Engineer

What You'll Be Doing :

Evolve your career by joining Evolent Health as an ML / LLM Operations Engineer. In this role you will monitor, evaluate, and ensure compliance for LLMs in healthcare settings, working across data science, engineering, and compliance teams to deliver reliable, regulatory-compliant AI services.

Collaboration Opportunities :

Partner with Data Science & Engineering, DevOps, Compliance, Quality Assurance, Clinical Support, and Product Management to keep AI systems operating reliably and meeting all healthcare sector requirements.

What You Will Be Doing :

Develop and maintain standardized evaluation frameworks to consistently measure LLM performance across relevant healthcare metrics.

Build monitoring systems using Logfire to track AI model performance, detect drift, and alert the team to anomalies.

Create testing infrastructure for prompt versions, model selection, and quality assurance processes.

Design and implement audit sampling processes for continuous quality monitoring and clinical review workflows.

Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare.

Optimize LLM operations through intelligent model selection, prompt engineering, and cost management strategies.

Support the transition from successful POCs to production-ready services with appropriate testing and validation.

Partner with DevOps on infrastructure requirements while focusing on AI-specific monitoring and optimization.

Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems.

Collaborate with Clinical Support Liaison to incorporate clinical feedback into system improvements.

Prepare regular reports on AI system quality, performance metrics, and compliance status.

Required Qualifications :

Bachelor's or master's degree in computer science, data science, or related field.

2+ years of experience with Python development and at least one production LLM implementation.

Strong proficiency in SQL for complex log analysis and metrics generation.

Demonstrated experience with LLM APIs and frameworks (experience with PydanticAI, LangChain, or similar).

Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting.

Understanding of LLM behavior, prompt engineering, and common failure modes in production.

Experience building evaluation or testing frameworks for AI / ML systems.

Strong communication skills for cross-functional collaboration.

Experience with healthcare AI applications and compliance requirements is preferred.

Familiarity with multiple LLM providers (OpenAI, Anthropic, Google, Azure) is preferred.

Knowledge of Pydantic ecosystem including PydanticAI and Logfire is preferred.

Understanding of LLM evaluation metrics and methodologies is preferred.

Experience building tools for non-technical users is preferred.

Basic knowledge of containerization (Docker) for local testing and development is preferred.

Experience with cloud environments (AWS, Azure) as a user is preferred.

Understanding of API rate limiting, quota management, and cost optimization strategies is preferred.

Knowledge of CI / CD concepts for ML model deployments is preferred.

Experience with regulatory compliance and audit processes is preferred.

Excellent documentation skills and attention to detail is preferred.

Evolent is an equal opportunity employer and considers all qualified applicants equally without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, or disability status.

The expected base salary / wage range for this position is $120,000 - $140,000. This position is also eligible for a bonus component that would be dependent on pre-defined performance factors. As part of our total compensation package, Evolent is proud to offer comprehensive benefits (including health insurance benefits) to qualifying employees. All compensation determinations are based on the skills and experience required for the position and commensurate with experience of selected individuals, which may vary above and below the stated amounts.

J-18808-Ljbffr