AI Engineer, Evaluation and Reliability

Senior Engineer, AI Evaluation and Reliability / Contract-to-Hire or Direct Hire / Redwood City / Hybrid, onsite 3 days per week / This position pays $70-80 / hr. W2 for Contract, $140-190K annually upon conversion / US Citizens and Green Card holders only Summary : Our client is looking for a Senior Engineer, AI Evaluation & Reliability to lead the design and execution of evaluation, quality assurance, and release gating for our agentic AI features. You'll develop the pipelines, datasets, and dashboards that measure and improve agent performance across real-world SOC workflows ensuring every release is safe, reliable, efficient, and production-ready. You will guarantee that our agentic AI features operate at full production scale, ingesting and active on millions of SOC alerts per day, with measurable impact on analyst productivity and risk mitigation. This role partners closely with the Product team to deliver operational excellence and trust in every AI-drive capability. Responsibilities : Define quality metrics : Translate SOC use cases into measurable KPI's (e.g., precision / recall, MTTR, false-positive rate, step success, latency / cost budgets). Build continuous evaluations : Develop offline / online evaluation pipelines, regression suites, and A / B or canary test; integrate them into CI / CD for release gating. Curate and manage datasets : Maintain gold-standard datasets and red-team scenarios; establish data governance and drift monitoring practices. Ensure safety, reliability, and explainability : Partner with Platform and Security Research to encode guardrails, policy enforcement, and runtime safety checks. Expand adversarial test coverage (prompt injection, data exfiltration, abuse scenarios). Ensure explainability and auditability of agent decisions, maintaining traceability and compliance of AI-driven workflows. Production reliability & observability : Monitor and maintain reliability of agentic AI features post-release define and uphold SLIs / SLOs, establish alerting and rollback strategies, and conduct incident post-mortems. Design and implement infrastructure to scale evaluation and production pipelines for real-time SOC workflows across cloud environments. Drive agentic system engineering : Experiment with multi-agent systems, tool-using language models, retrieval-augmented workflows, and prompt orchestration. Manage model and prompt lifecycle track version, rollout strategies, and fallbacks; measure impact through statistically sound experiments. Collaborate cross-functionally : Work with Product, UX and Engineering to prioritize high-leverage improvements, resolve regressions quickly, and advance overall system reliability. Required Skills : 6+ years building evaluation or testing infrastructure for ML / LLM systems or large-scale distributes system Proven ability to translate product requirements into measurable metrics and test plans. Strong Python skills Strong Experience with modern data tooling Hands-on experience running A / B tests, canaries, or experiment frameworks. Experience defining and maintaining operational reliability metrics (SLIs / SLOs) for AI-driven systems. Familiarity with large-scale distributed or streaming systems serving AI / agent workflows (millions of events or alerts / day). Excellent communication skills able to clearly convey technical results and trade-offs to engineer, PMs, and analysts. Pay for this position is based on market location and may vary depending on job-related knowledge, skills, and experience. As a contractor you may also be eligible for health benefits such as health, dental, and vision as well as access to a 401K plan. A sign-on payment and restricted stock units may be provided as part of the compensation package, in addition to a full range of medical, financial, and / or other benefits, dependent on the position offered by our client. Applicants should apply via The Mice Groups Inc. website (www.micegroups.com) or through this careers site posting. We are an equal opportunity employer and value diversity at The Mice Groups Inc. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records. Pursuant to the Los Angeles Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records. The Mice Groups Inc. values your privacy. Please consult our Candidate Privacy Notice, for information about how we collect, use, and disclose personal information of our candidates. Privacy Policy One of the basic principles The Mice Groups follows in designing and operating this website is that we ask for only the information we need to provide the service you’ve requested. The Mice Groups does not currently collect personal identifying information via its website except (i) to the extent that you provide this information in an online job application and (ii) to the extent that your web browser provides personal identifying information. The Mice Groups will use your personally identifying information solely for the purpose for which you submitted the information. The Mice Groups may, however, aggregate certain elements of your personal identifying information with the information of other users of our website to analyze the usefulness and popularity of various web pages on its website. The Mice Groups reserves the right to change this policy at any time by posting a new privacy policy at this location. Questions regarding this statement should be directed to info@micegroups.com