Job Description
Job Title: Lead Data Engineer – Databricks
Job Summary
We are seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud-based lakehouse platforms. The role combines hands-on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.
Key Responsibilities
Technical Leadership & Architecture
Lead the design and implementation of Databricks Lakehouse architectures
Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
Drive architectural decisions for batch and streaming data pipelines
Establish coding standards, best practices, and reusable frameworks
Data Engineering & Databricks
Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
Optimize Spark jobs for performance, reliability, and cost
Implement Delta Lake features (ACID, time travel, schema enforcement)
Develop and manage Databricks workflows, jobs, and clusters
Cloud & Platform Integration
Architect Databricks solutions on Azure (preferred) or AWS
Integrate Databricks with cloud storage and data services
Azure: ADLS, ADF, Synapse
AWS: S3, Glue, Redshift
Enable BI and analytics consumption (Power BI, Tableau)
Governance, Security & DevOps
Implement data governance using Unity Catalog
Define RBAC, data access controls, and security best practices
Enable CI/CD for Databricks using GitHub / Azure DevOps
Use Infrastructure-as-Code (Terraform) for environment management
Leadership & Collaboration
Lead, mentor, and grow data engineering teams
Conduct design and code reviews
Collaborate with Data Architects, Product Owners, and stakeholders
Support production releases, monitoring, and incident resolution
Required Skills
Databricks & Big Data
Expert-level Databricks experience (Azure or AWS)
Strong Spark / PySpark / Spark SQL expertise
Delta Lake and Lakehouse architecture
Streaming (Structured Streaming) experience
Cloud & Data Platforms
Strong experience with Azure or AWS cloud platforms
Data orchestration tools (ADF, Airflow, or similar)
Strong SQL and data modeling skills
DevOps & Automation
Git-based version control
CI/CD pipelines for data engineering workloads
Terraform or similar IaC tools
Preferred Qualifications
Experience with MLflow and MLOps workflows
Exposure to Microsoft Fabric or Snowflake
Databricks certifications (Professional Data Engineer / Architect)
Experience working in Agile environments
Education
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
Quick Fit Indicators
Lead Data Engineer- DataBricks • Milpitas, California, United States
Position: Data Engineer II Location: Sunnyvale, California Duration: Contract Job ID: 175305 Job Overview: We are seeking a skilled and detail-oriented Data Engineer II to join our team.The ideal c... Show more
Location: Cupertino, CA (Hybrid).We are seeking a Data Engineer with strong data pipeline development skills and hands-on experience managing containerized workflows in Kubernetes and Docker.This r... Show more
Leadgenius is a leader in SaaS for Enterprise Sales and Marketing teams.We have a full service WebApp, Chrome plugin and data service that delivers high quality sales and marketing intelligence to ... Show more
A fast-growing healthcare data company in California seeks a talented Data Engineer to enhance their data platform.The role demands 5+ years of experience in Data Engineering, expertise in ETLs, bi... Show more
Talent Acquisition Specialist | Recruitment Manager.Helping US/Canada Based Fortune 100/500 Clients with their IT/Non-IT Staffing (Contract).Proficiency in Python scripting and development.Strong u... Show more
Client is looking for Big Data engineers with expert level experience in developing enterprise software applications on Spark/Hadoop, NoSql, InMemory Databases.You will be part of the team building... Show more
Title: Data and Cloud Solutions Architect Location: San Jose, CA (100% on-site) Pay: $65/hr Job Description: The Data and Cloud Solutions Architect will be a key contributor to designing, evolving,... Show more
The application window is expected to close on: 06/30/2026.Job posting may be removed earlier if the position is filled or if a sufficient number of applications are received.The SaaS Engineering o... Show more
A leading technology company in California is seeking a skilled Data Engineer to manage data warehouse plans and ensure data availability.This role involves collaborating with engineers and product... Show more
CoreWeave is the top-rated AI‑cloud for high-performance GPU infrastructure across AI/ML, visual effects, rendering, and real-time inference.Our stack is engineered for speed, scale, and cost‑effic... Show more
Join us in building the machine learning platform that enables teams at Apple to build Apple Intelligence and many other intelligent experiences across hardware, software and service products.As a ... Show more
TEKFORTUNE INC is seeking a Senior Azure Data Engineer in Fremont, California, with over 10 years of experience in IT infrastructure and cloud engineering.The ideal candidate will have 6+ years of ... Show more
JPMorgan Chase in Palo Alto is seeking a Data Architect III to lead architecture solutions in the Consumer & Community Banking sector.This role includes responsibilities for developing high-quality... Show more
A major semiconductor company in San Jose, CA is looking for an experienced Data Engineer to design and maintain an enterprise-wide Big Data and Data Analytics Platform.Responsibilities include man... Show more
Build and maintain efficient and reliable data pipelines to support business initiatives in the user growth domain.Drive data warehouse design in accordance to long-term product goals.Work closely ... Show more
This range is provided by DynPro Inc.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Direct message the job poster from DynPro Inc.Tech Recruiti... Show more
A global consultancy firm is seeking a highly skilled Senior Consultant Data Engineer to design and build analytics solutions in a cloud environment.You will work with technologies like Databricks ... Show more
A technology solutions provider is seeking a Senior Data Management Engineer for its San Jose, CA location.The role involves analyzing customer requirements, developing Data Management applications... Show more
Early‑stage, cybersecurity investment (valued over $100M at Seed), founded by a successful serial entrepreneur, is looking to hire a Founding Data Engineer.Bonus points for prior industry experienc... Show more
A cybersecurity company in Santa Clara is seeking a Principal Engineer to design and develop groundbreaking SASE technologies.The ideal candidate will have over 10 years of experience with system s... Show more
