Automation Tester (Playwright)

Position Title: Automation Tester (with Playwright)
Start Date: ASAP
Approved On-Site Location(s): Irving, TX
On-Site Schedule: Hybrid (3 days/week in office)
# Of Openings: 1

Skillset: API, UI, Performance Testing, Playwright

Overview:
The IAM Tools Engineering team (ITE) is looking for a highly motivated and skilled AI Test Engineer to join our growing team. The AI Test Engineer will be responsible for designing, developing, and executing comprehensive test strategies and plans for AI/ML models and applications. This role requires a strong understanding of AI principles, machine learning workflows, data quality, and various testing methodologies specific to intelligent systems. The ideal candidate will work closely with the ITE engineering team and application users, have related experience developing tests in an iterative manner, be passionate about quality, possess excellent problem-solving skills, and thrive in a collaborative, fast-paced environment. This person must have a proven track record of demonstrating their ability to effectively design and develop test frameworks, execute tests manually and in an automated manner, and have a strong technical focus. Critical success factors will include the ability to effectively engage in a matrixed organization, develop partnerships with many business, analyst, and development teams, and drive the quality assurance strategy for a cutting-edge AI enable self-service identity solution.

Responsibilities include:

• Develop and implement robust test plans and strategies tailored for AI/ML systems, including defining test objectives, methodologies, and scenarios.
• Review design specifications to understand the scope, requirements, and requested functions of the software product, partnering with the team during delivery of the stories to ensure stories are accurately implemented
• Identify appropriate parameters, functions, and data to test and validate requirements and acceptance criteria; ensuring a successful development and deployment that meets the business needs
• Ability to create effective granular and end to end test scenarios for user interface, data transactions, integrated systems, and functional aspects of web-based software applications
• Collaborate with engineers, and product managers to understand AI model requirements, use cases, and potential risks.
• Define quality metrics for AI models, focusing on aspects like accuracy, fairness, robustness, and interpretability.
• Design and create detailed test cases for various types of AI testing, including: Validating AI model outputs against expected behavior, prompt-response accuracy, and contextual understanding.
• Assessing model speed, latency, throughput, and scalability under different load conditions.
• Bias and Fairness Testing: Identifying and mitigating biases in training data and model predictions to ensure ethical and equitable AI behavior.
• Evaluating model resilience against various inputs, including malicious or unexpected data.
• Data Quality Testing: Ensuring the integrity, consistency, relevance, and representativeness of data used for training and testing AI models.
• Identifying vulnerabilities in AI systems and ensuring compliance with data protection standards (e.g., GDPR, CCPA).
• Verifying seamless integration of AI models with other systems, APIs, and existing software.
• Execute manual and automated test cases, analyzing results and identifying defects, anomalies, and inconsistencies.
• Implement and maintain automated testing frameworks and tools specifically designed for AI/ML models and pipelines
• Develop test scripts using programming languages commonly used in AI/ML (e.g., Python, Java, JavaScript).
• Utilize AI-specific testing tools and frameworks to enhance testing efficiency and accuracy.
• Integrate testing processes into Continuous Integration/Continuous Deployment (CI/CD) pipelines.
• Accurately document, track, and prioritize defects and issues found during testing.
• Work closely with development teams to ensure timely resolution of identified problems.
• Prepare comprehensive test reports, providing actionable insights into AI model performance and quality.
• Work with leaders, stakeholders, SMEs, scrum team, project managers, etc. to ensure success against stated objectives, ensuring the team is delivering a successful product
• Collaborate with internal and external subject matter experts, developers, and other QA Engineers
• Produce effective test scenarios to capture and communicate business process and requirements
• Ability to communicate daily status, along with defects/issues found, at daily standups.

Required Qualifications:

• 5+ years of practical Software Testing experience with web-based applications using automated test engineering tools.
• College degree in Computer Science/Engineering or equivalent work experience
• Strong technical background required. Must have prior programming experience and solid understanding of how to embed test automation in CI/CD pipelines(e.g., Jenkins, GitHub Actions). Must also have experience in all types of testing, including unit, integration, system, functional, non-functional and performance testing.
• Understanding of cloud platforms (AWS, Azure, GCP) and their AI/ML services is a plus.
• Experience with various testing methodologies (functional, performance, security, etc.).
• Hands-on experience with automated testing frameworks.
• Strong proficiency in at least one programming language essential for AI/ML and test automation (e.g., Python).
• Understanding of AI/ML concepts, algorithms, and workflows (e.g., supervised/unsupervised learning, neural networks, natural language processing, computer vision).
• Experience with data validation, cleansing, and quality assurance processes for large datasets.
• Knowledge of API testing tools (e.g., Postman).
• Familiarity with version control systems (e.g., Git)
• Hands-on experience with planning, designing, executing tests and knowledge of performance and stress testing
• Experience implementing and maintaining quality test automation for web applications

Desired Qualifications:

• Experience being creative and having initiative to improve product quality coverage and effectiveness
• Strong analytical skills with high attention to detail and accuracy
• Strong organizational, multi-tasking, and prioritizing skills
• Experience with Agile methodologies (Ceremonies, Definition of Done, Definition of Ready, Lean Thinking, etc..)
• Experience with Selenium, Vulcan, SpecFlow, or similar testing tools
• Knowledge of DevOps practices
• Ability to work effectively in virtual environment where key team members and partners are in various time zones and locations
• Experience with Atlassian products (e.g. Confluence and JIRA)
• Experience with QA process and methodologies
• Experience communicating with technical and non-technical stakeholders across all levels of the organization
• Experience thriving in an ambiguous environment that demands constant self-prioritization”