Data Scientist Generative AI & Model Evaluation

Location :

4910 Tiedeman Road, Brooklyn Ohio

Position Location Policy

General Location : Open to candidates within the United States.

Hybrid Requirement (if within specific cities) : If the selected candidate resides in Cleveland, OH, Buffalo, NY, or Albany, NY, they are expected to work on-site 2 days per week at the nearest KeyBank office (non-branch location).

Remote Option : If the selected candidate lives outside of KeyBanks geographic footprint, the position will be considered fully remote.

Available for East Coast meetings, required

Within the Commercial Bank, the Data Scientist role is primarily responsible for identifying, developing, and critically evaluating AI-powered solutions across a range of use-cases spanning sales enablement, marketing content generation, enhanced customer onboarding experiences, intelligent servicing, and novel risk assessment methodologies. Leveraging current and emerging Generative AI technologies, particularly Large Language Models (LLMs), this role will focus on prompt engineering, data curation for LLM training / fine-tuning, evaluation methodology and technique selection for generative outputs, bias and fairness assessment of AI-generated content, robust documentation of generative model behavior and limitations, and continuous monitoring of AI application performance and ethical implications.

ESSENTIAL JOB FUNCTIONS

• Perform a broad range of quantitative works, including designing, developing, and implementing generative AI solutions, with a strong emphasis on prompt engineering and fine-tuning of LLMs to address business needs while adhering to model risk and regulatory requirements.
• Conduct ad hoc analysis of generative model outputs to identify areas for improvement.
• Research, compile, and evaluate large sets of data, focusing on suitability for LLM training, fine-tuning, and importantly, for robust evaluation of LLM outputs. This includes creating and curating ground truth datasets and adversarial examples for performance assessment.
• Develop and maintain internal frameworks for evaluating LLM performance, including metrics for accuracy, fluency, coherence, factual correctness, toxicity, bias, and safety. Test and configure vendor-provided LLM solutions, assessing their suitability and performance against defined criteria.
• Document LLM prompt engineering strategies, model configurations, evaluation methodologies, and observed behaviors, including limitations and potential failure modes. Support model validation by providing clear documentation on how the LLM functions and the results of its evaluations.
• Employ innovative techniques to drive continuous improvements in LLM output quality, relevance, and safety. Focus on techniques such as reinforcement learning from human feedback (RLHF), prompt optimization, and bias mitigation strategies to enhance effectiveness and efficiency, e.g., improving factual accuracy and reducing harmful outputs.
• Proactively develop and build technical skills and business knowledge, particularly in Generative AI ethics, responsible AI principles, and LLM-specific risk management. Effectively collaborate with compliance, technology, and risk partners to ensure the safe and ethical deployment of AI applications.
• Delivers clear, persuasive communication tailored to stakeholders; proactively shares relevant information and excels in high-stakes or conflict-driven conversations.
• Builds strong relationships and collaborates effectively; consults with mid-level leaders to resolve key issues and influence outcomes.
• Leverages deep banking and financial insight to drive data-informed strategies aligned with business goals and market dynamics

EDUCATION

WORK EXPERIENCE

PREFERRED QUALIFICATIONS

PHYSICAL DEMANDS

TRAVEL REQUIREMENTS

Key has implemented a role-based Mobile by Design approach to our employee workspaces, dedicating space to those whose roles require specific workspaces, while providing flexible options for roles which are less dependent on assigned workspaces and can be performed effectively in a mobile environment. As a result, this role may be mobile or home based, which means you may work either at a home office or in a Key facility to perform your job duties.

COMPENSATION AND BENEFITS

This position is eligible to earn a base salary in the range of $79,000 to $85,000 annually depending on location and job-related factors such as level of experience. Compensation for this role also includes eligibility for short-term incentive compensation and deferred incentive compensation subject to individual and company performance.

Please click here for a list of benefits for which this position is eligible.

Key has implemented an approach to employee workspaces which prioritizes in-office presence, while providing flexible options in circumstances where roles can be performed effectively in a mobile environment.

Job Posting Expiration Date : 10 / 08 / 2025KeyCorp is an Equal Opportunity Employer committed to sustaining an inclusive culture. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or veteran status.

Qualified individuals with disabilities or disabled veterans who are unable or limited in their ability to apply on this site may request reasonable accommodations by emailing

LI-Remote

#J-18808-Ljbffr