Databricks Lead Engineer

Job Title : Databricks Lead Engineer

Job Type : Contract

Duration : 6 Months

100% Remote role

Job Summary :

We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our Databricks platform. As a technical leader, you will leverage your expertise in cloud technologies and data engineering to ensure high performance and scalability of our data solutions. This position requires a passion for data, a commitment to innovation, and a focus on delivering efficient and effective solutions.

Key Responsibilities :

• Provide technical guidance to a team of data engineers in the development and maintenance of scalable data pipelines for the platform.
• Optimize data capture, ingestion, and processing workflows to enable advanced analytics, data science, and operational intelligence.
• Collaborate with Data Platform Architects and other Data Engineers to integrate feature stores, manage model deployment, and implement data preparation workflows.
• Ensure data engineering processes are designed for high performance, scalability, and resource efficiency.
• Provide technical leadership and mentorship to team members, fostering an environment of continuous improvement and innovation.
• Design and maintain data pipelines for streaming data, batch processing, and real-time analytics.
• Implement secure and scalable data capture and ingestion workflows.
• Support data preparation, exploration, and visualization to facilitate data-driven decision-making.
• Design and implement robust data quality frameworks with Databricks ingestion pipelines, ensuring reliable, accurate, and trustworthy data for downstream analytics and decision making.
• Align and parameterize Databricks ingestion pipelines with CI / CD standards
• Design and implement conceptual, logical, and physical data models using best practices for Lakehouse architecture
• Work closely with other engineering teams to ensure data pipelines are secure, efficient, and aligned with business requirements.
• Lead client facing workshops.
• Document Use Cases
• Define and implement standards and best practices for the Databricks platform

Qualifications :