Databricks Technical Lead Level 3

Job Description

Job Description

Location - Cincy or Charlotte, hybrid but could be in office 4 days / week

Job Summary :

We are seeking a seasoned Databricks Technical Lead tojoin our HR Systems Engineering team. This role is pivotal in enhancing theexperience of over 420,000 associates by leading the design, build, andoptimization of our data platform, services, APIs, and cloud migrations.

As a product-centric role within an agile deliveryframework, you will ensure our data solutions align with business objectivesand deliver tangible value. Join us in an organization recognized as one of thebest places to work in IT for seven consecutive years.

Required Qualifications :

10+ years of experience in data engineering, big data, orAPI development, with at least 3+ years in a leadership role.

Proven experience leading product-centric dataengineering initiatives in an agile delivery environment.

Expertise in Azure Databricks, Apache Spark, Azure SQL,and other Microsoft Azure services.

Strong programming skills in Python, Scala, and SQL fordata processing and API development.

Experience in building and managing APIs (REST, GraphQL,gRPC) and microservices.

Hands-on experience with Azure Data Factory (ADF), AzureSynapse Analytics, and Delta Lake.

Proficiency in CI / CD pipelines, Terraform / Bicep, andInfrastructure-as-Code.

Experience with data security and compliance measures(e.g., encryption, access control, auditing) for sensitive HR and employeedata.

Strong problem-solving skills, with a focus onperformance tuning, security, and cost optimization.

Experience with containerization (Docker, Kubernetes) andevent-driven architecture is a plus.

Exposure to Informatica for ETL / ELT and data integration.

Excellent communication and leadership skills in afast-paced environment.

This is a contract-to-hire position, and candidates mustbe prepared to transition to a full-time employee role.

Preferred Qualifications :

Microsoft Certified : Azure Solutions Architect Expert orDatabricks Certified Data Engineer / Architect certification.

Experience with agile development methodologies such asScrum or SAFe.

Familiarity with machine learning workflows in AzureDatabricks.

Knowledge of Azure API Management and Event Hub for APIintegration.

Experience with Informatica PowerCenter or InformaticaIntelligent Cloud Services (IICS).

Hands-on experience with Oracle HCM database models andAPIs, including integrating this data into enterprise data solutions

Experience with HR Analytics.

Proximity to our Cincinnati, OH, or Charlotte, NC,office, with the ability to work in a hybrid model (in-office Monday toThursday).

Key Responsibilities

Core Responsibilities :

Develop and maintain the HR data domain in a secure,compliant, and efficient manner in accordance with best practices.

Lead the development of a data engineering teamresponsible for designing scalable, high performance data solutions, APIs, andmicroservices in Azure Databricks, Azure SQL and Informatica.

Ensure the highest levels of security and privacyregarding sensitive data.

This is a job for an exceptional professional who deeplyunderstands big data processing, data architecture, cloud migrations, APIdevelopment, data security, and agile methodologies in the Azure ecosystem.

Key Responsibilities :

Azure Databricks & Big Data Architecture :

Design and implement scalable data pipelines andarchitectures on Azure Databricks.

Optimize ETL / ELT workflows, ensuring efficiency in dataprocessing, storage, and retrieval.

Leverage Apache Spark, Delta Lake, and Azure-nativeservices to build high-performance data solutions.

Ensure best practices in data governance, security, andcompliance within Azure environments.

Troubleshoot and fine-tune Spark jobs for optimalperformance and cost efficiency.

Azure SQL & Cloud Migration :

Lead the migration of Azure SQL to Azure Databricks,ensuring a seamless transition of data workloads.

Design and implement scalable data pipelines to extract,transform, and load (ETL / ELT) data from Azure SQL into Databricks Delta Lake.

Optimize Azure SQL queries and indexing strategies beforemigration to enhance performance in Databricks.

Implement best practices for data governance, security,and compliance throughout the migration process.

Work with Azure Data Factory (ADF), Informatica, andDatabricks to automate and orchestrate migration workflows.

Ensure seamless integration of migrated data with APIs,machine learning models, and business intelligence tools.

Establish performance monitoring and cost- optimizationstrategies post-migration to ensure efficiency.

API & Services Development :

Design and develop RESTful APIs and microservices forseamless data access and integrations.

Implement scalable and secure API frameworks to exposedata processing capabilities.

Work with GraphQL, gRPC, or streaming APIs for real-timedata consumption.

Integrate APIs with Azure-based data lakes, warehouses,Oracle HCM, and other enterprise applications.

Ensure API performance, monitoring, and security bestpractices (OAuth, JWT, Azure API Management).

HR Data Domain & Security :

Build and manage the HR data domain, ensuring a scalable,well-governed, and secure data architecture.

Implement role-based access control (RBAC), encryption,and data masking to protect sensitive employee information.

Ensure compliance with GDPR, CCPA, HIPAA, and other dataprivacy regulations.

Design and implement audit logging and monitoring totrack data access and modifications.

Work closely with HR and security teams to define dataretention policies, access permissions, and data anonymization strategies.

Enable secure API and data sharing mechanisms for HRanalytics and reporting while protecting employee privacy.

Work with Oracle HCM data structures and integrate themwithin the Azure Databricks ecosystem.

Product-Centric & Agile Delivery :

Drive a product-centric approach to data engineering,ensuring alignment with business objectives and user needs.

Work within an agile delivery framework, leveragingScrum / Kanban methodologies to ensure fast, iterative deployments.

Partner with product managers and business stakeholdersto define data-driven use cases and prioritize backlog items.

Promote a continuous improvement mindset, leveragingfeedback loops and data-driven decision-making.

Implement DevOps and CI / CD best practices to enable rapiddeployment and iteration of data solutions.Leadership & Collaboration :

Provide technical leadership and mentorship to a team ofdata engineers and developers.

Collaborate closely with business stakeholders, productmanagers, HR teams, and architects to translate requirements into actionabledata solutions.

Advocate for automation, DevOps, andInfrastructure-as-Code (Terraform, Bicep) to improve efficiency.

Foster a culture of innovation and continuous learningwithin the data engineering team.

Stay updated on emerging trends in Azure Databricks,Azure SQL, Informatica, Oracle HCM, and cloud technologies.

Note to Vendors

Top 3 Skills - Databricks certified technical lead,Python, Data Domain experience

Project - hasrecently transitioned from MS Azure to Databricks. They are in the infancystage of setting up data domains and therefore need an expert in this area.They are looking for someone who is an expert in Databricks to lead this team.

Scope - There is 3+ years of initial work to do so thisposition has long term potential

Soft Skills - Strong ability to lead and communicateacross several technical teams in an agile environment; on and offshore.Transparency and sticking to timelines are also important soft skills.

Max Rate - I am not setting an official max in the systembut please be cognizant of rate. I will reject people if they come in over

Prescreening - This prescreening will consist of aHireVue with multiple choice questions, not a video.

This need is asap and they will interview and makedecisions quickly.