Data Engineer

We are seeking a highly skilled Data Engineer to design, develop, and optimize batch ETL pipelines on Databricks within a cloud-based Lakehouse architecture. The role focuses on ingesting high volumes of banking and financial data, ensuring data integrity, governance, and performance for analytics, regulatory, and machine learning use cases. The ideal candidate brings hands-on experience with Apache Spark, Delta Lake, and Azure cloud data services, along with a deep understanding of data compliance and banking regulations.

Job Responsibilities :

• Build and Integrate Data Pipelines : Design and implement batch ETL pipelines using Databricks (PySpark, Spark SQL) to support a scalable Lakehouse infrastructure.
• Data Quality and Integrity : Ensure high data quality through validation, cleansing, enrichment, and versioning using Delta Lake ACID features.
• Performance Optimization : Tune Spark jobs, optimize partitioning, and manage dependencies to meet SLAs for large-scale data processing.
• Governance and Compliance : Enforce data security and compliance policies (e.g., encryption, data lineage, access control) per banking regulations (AML, BSA).
• Cross-Team Collaboration : Partner with data architects, analysts, BI, risk, and data science teams to deliver reliable and insightful data pipelines.
• Continuous Improvement : Automate and enhance pipeline workflows using tools such as Databricks Jobs, Delta Live Tables (DLT), and Airflow.
• Support Auditability : Maintain historical data and change tracking for audit and compliance requirements.
• Other Duties : Adhere to federal laws and regulations and perform other duties as assigned.

Required Skills :

Preferred Skills :

Certifications :

Azure / AWS / GCP Data Engineer Certifications (Preferred)

J-18808-Ljbffr