Azure Databricks Engineer

Azure Databricks Engineer

Remote with occasional travel to Chicago, IL to attend workshops

Key responsibilities

Data pipeline development :

Design, build, and manage scalable data pipelines for both batch and streaming data using tools like Azure Data Factory , Databricks notebooks, and Delta Lake .

Solution architecture :

Develop the architecture for data engineering solutions, integrating various Azure services such as Azure Databricks, Azure Data Lake Storage, and Azure Synapse Analytics.

ETL / ELT processes :

Compose and orchestrate data transformation logic using languages like Python (PySpark), Scala, and SQL.

Data management and governance :

Implement data governance and security using tools like Unity Catalog and Azure Key Vault to manage access and secrets.

Performance optimization :

Monitor and optimize Databricks clusters, jobs, and data solutions for high performance and cost-efficiency.

Collaboration :

Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

Required skills and technologies

• Core platform : Azure Databricks and its components.
• Programming languages : Python (with PySpark) is often a primary requirement, along with Scala and SQL.
• Data processing : Experience with distributed data processing frameworks, especially Apache Spark, is crucial.
• Azure services : Familiarity with other Azure data services, including Azure Data Factory , Azure Data Lake Storage , Azure SQL Database , and Azure Synapse Analytics .
• Data Lakehouse concepts : Understanding of Lakehouse architecture and the use of Delta Lake for data storage and management.
• CI / CD : Experience in building and managing CI / CD pipelines for data environments.
• Data visualization : Connecting to Databricks tables from visualization tools like Power BI .