AI Engineer

Job Description:

• Continuous LLM Evaluation: Design and operate a systematic, ongoing process to evaluate new and emerging LLMs across accuracy, relevancy, speed, and cost — continuously benchmarking them against the specific tasks in our orchestration pipeline proactively optimizing outcomes.
• Eval Framework Development: Build and maintain rigorous evaluation frameworks (Evals) to measure LLM output accuracy, relevance, faithfulness, and speed with a specific focus on reducing hallucinations in medical record summarization and legal document analysis.
• Proactive Model Transition Planning: Monitor the LLM landscape across providers to identify deprecation timelines and suitable replacement models — and own the full execution of those transitions, including integrating new models into the production pipeline and maintaining necessary changes to account for model behavior.
• AI Pipeline Optimization: Directly implement optimizations to LLM-based orchestration pipelines for document understanding, medical record summarization, case chronology generation, and drafting support — owning code changes, deployments, and production validation from start to finish.
• Cross-Functional Collaboration: Partner with product and GTM stakeholders to communicate model evaluation findings — then lead the technical implementation yourself rather than delegating execution to a separate engineering team.
• End-to-End Implementation Ownership: Take full responsibility for shipping model changes into production — writing the integration code, managing deployments, running validation tests, and ensuring a clean rollout.
• Operational Monitoring: Implement monitoring and observability for model performance in production, benchmarking outputs and cost, detecting drift with ongoing and continuous reporting to management.
• Documentation: Maintain thorough documentation of evaluation methodologies, model comparison results, transition decisions, and runbooks for the systems you own.

Requirements:

• 5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying large language models in production environments — including building and deploying the models to cloud (AWS or GCP) infrastructure at scale.
• Hands-on development and implementation of multiple RAG solutions.
• Hands-on experience leveraging embedding models and vector databases.
• Hands-on experience building agentic workflows.
• Deep familiarity with the LLM ecosystem and the ability to critically assess model capabilities, limitations, and fit for specific tasks, including cost, quality, speed, and capability tradeoffs.
• Proven experience designing and operating evaluation frameworks to measure LLM output quality, including accuracy, relevancy, and hallucination detection in high-stakes domains (legal, medical, or similar).
• Strong software engineering foundation with proven experience writing production-deployed solutions, including LLM orchestration frameworks and multi-model pipelines.
• Comfort working in a fast-paced, high-ambiguity environment with strong ownership, tight feedback loops, and a bias for systematic process-building over one-off fixes.
• Excellent communication skills; ability to translate complex model evaluation findings into clear recommendations for engineering, product, and non-technical stakeholders.