Microsoft Azure Data Engineer

Microsoft Azure~Digital

6-9 yrs

Data Engineer - Databricks

• Must have at least 8+ years of extensive Data Engineering Experience in Databricks, ETL / ELT using Data pipelines, SQL / Procedure experience.
• Must have 8+ Years of hands-on development proficiency in implementing Databricks solutions using Scala and Spark with Data frames and Notebooks / SQL.
• Must have 8+ Years of hands-on development and performance tuning / enhancement experience in Scala / PySpark and SQL.
• Must have 8+ Years of hands-on development experience in working on Azure Cloud.
• Must have strong experience with Databricks, including developing and optimizing Spark jobs, data transformations, and data processing workflows.
• Must have experience and a good understanding on the below topics

Databricks Delta Lake storage

Unity Catalog

Autoloader

Lakehouse Architecture

Medallion Architecture

Databricks Serverless Options.

Dataset and Data frame

Databricks Clusters Configuration and Sizing

Databricks Job Orchestration

Spark Structured Streaming