GenAI / LLM Systems Test Engineering Manager

Pearson Engineering Manager

Pearson is seeking a highly technical GenAI / LLM Systems Test Engineering Manager to lead quality engineering efforts for the XL+ initiativea flagship program delivering next-generation AI-powered learning solutions such as the Personal Teaching Assistant (PTA), Personal Learning Assistant (PLA), and Open Educational Resource (OER) support. This role goes beyond traditional quality management. We are looking for a technical leader with deep expertise in AI / ML systems, Large Language Models (LLMs), and modern test engineering practices. The ideal candidate will design and drive innovative test strategies for AI solutions, ensuring accuracy, reliability, scalability, and trust in our GenAI-powered educational products. You will work closely with development, product, and data science teams, while guiding QE engineers to adopt new frameworks and approaches for testing AI workflows, prompt outputs, and integrated learning systems.

Key Responsibilities :

• Define and drive AI Test Strategy :
• Create comprehensive test strategies specifically for GenAI products and LLM-based workflows, including model accuracy, bias, safety, and reliability validation.
• Develop and maintain AI testing frameworks, including prompt testing and evaluation of model outputs
• Hands-on Technical Leadership :
• Build and evolve test automation frameworks, pipelines, and evaluation harnesses for AI / ML models, APIs, and integrated systems.
• AI-Specific Quality Validation :
• Design tests around prompt engineering, hallucination detection, model evaluation metrics, and edge-case scenarios.
• System & Integration Testing :
• Validate end-to-end workflows, including multi-agent orchestration, MCP models, APIs, and UI interactions.
• Work closely with Performance & Scalability Test tesm to ensure AI-driven systems perform consistently at scale across different use cases and data sets.
• Cross-Functional Collaboration :
• Partner with Dev, Product, and Data Science to align quality standards with evolving GenAI architectures.
• Mentorship :
• Grow and mentor a team of QE engineers, building technical depth and expanding the team's expertise in AI testing.
• Continuous Improvement :
• Drive defect analysis, implement lessons learned, and adopt emerging tools and methods for testing AI.
• Coordinate testing activities across globally distributed QE teams
• Manage defect tracking, root cause analysis, and continuous improvement for all releases
• Mentor and support a growing team of QA engineers

Qualifications :

6+ years of experience in Quality Engineering, with at least 3 years in a technical leadership or managerial role.

Strong technical expertise in AI / ML product testing, including GenAI workflows, model evaluation, and / or LLM-based solutions.

Hands-on experience with test automation frameworks, API / UI testing, and CI / CD pipelines.

Familiarity with LLM testing challenges (e.g., prompt variation, non-deterministic outputs, evaluation metrics, model drift).

Knowledge of Python or similar languages for building test tools or harnesses.

Experience in performance, scalability, and security testing of distributed systems.

Exposure to prompt engineering, bias testing, or AI ethics frameworks is a plus.

Bachelor's degree in Computer Science, Engineering, or related field (advanced degree preferred).

Proven ability to work in cross-functional, Agile / Scrum environments.

Excellent communication and leadership skills.

Bachelor's degree in Computer Science, Engineering, or related field; advanced degree is a plus.

Why Join Us?

This is an exciting opportunity to be at the forefront of educational technology innovation. You will play a critical role in shaping the quality of AI-driven learning tools that impact students and instructors worldwide. Join us as we modernize digital learning and drive meaningful transformation in education.

Compensation at Pearson is influenced by a wide array of factors including but not limited to skill set, level of experience, and specific location. As required by the California, Colorado, Hawaii, Illinois, Maryland, Minnesota, New Jersey, New York State, New York City, Vermont, Washington State, and Washington DC laws, the pay range for this position is as follows : The minimum full-time salary range is between $165,000 - $170,000 - This position is eligible to participate in an annual incentive program, and information on benefits offered is here. Applications will be accepted through until the 5 November 2025. This window may be extended depending on business needs.

At Pearson, our purpose is simple : to help people realize the life they imagine through learning. We believe that every learning opportunity is a chance for a personal breakthrough. We are the world's lifelong learning company. For us, learning isn't just what we do. It's who we are. To learn more : We are Pearson. Pearson is an Equal Opportunity Employer and a member of E-Verify. Employment decisions are based on qualifications, merit and business need. Qualified applicants will receive consideration for employment without regard to race, ethnicity, color, religion, sex, sexual orientation, gender identity, gender expression, age, national origin, protected veteran status, disability status or any other group protected by law. We actively seek qualified candidates who are protected veterans and individuals with disabilities as defined under VEVRAA and Section 503 of the Rehabilitation Act. If you are an individual with a disability and are unable or limited in your ability to use or access our career site as a result of your disability, you may request reasonable accommodations by emailing TalentExperienceGlobalTeam@grp.pearson.com.