Data Engineer

Job Description

Job Description

Our client is undergoing a major digital transformation, shifting toward a cloud-native, API-driven infrastructure. They’re looking for a Data Engineer to help build a modern, scalable data platform that supports this evolution. This role will focus on creating secure, efficient data pipelines, preparing data for analytics, and enabling real-time data sharing across systems.

As the organization transitions from older, legacy systems to more dynamic, event-based and API-integrated models, the Data Engineer will be instrumental in modernizing the data environment—particularly across the bronze, silver, and gold layers of their medallion architecture.

Key Responsibilities :

• Design and deploy scalable data pipelines in Azure using tools like Databricks, Spark, Delta Lake, DBT, Dagster, Airflow, and Parquet.
• Build workflows to ingest data from various sources (e.g., SFTP, vendor APIs) into Azure Data Lake.
• Develop and maintain data transformation layers (Bronze / Silver / Gold) within a medallion architecture.
• Apply data quality checks, deduplication, and validation logic throughout the ingestion process.
• Create reusable and parameterized notebooks for both batch and streaming data jobs.
• Implement efficient merge / update logic in Delta Lake using partitioning strategies.
• Work closely with business and application teams to gather and deliver data integration needs.
• Support downstream integrations with APIs, Power BI dashboards, and SQL-based reports.
• Set up monitoring, logging, and data lineage tracking using tools like Unity Catalog and Azure Monitor.
• Participate in code reviews, design sessions, and agile backlog grooming.

Additional Technical Duties :

Tech Stack :

Skills & Competencies :

Qualifications :