Sr Data Engineer - Data Platform

Summary:

We are seeking a Sr. Data Engineer Data Platform for our Data Science & Insights Department.

The Senior Data Engineer - Data Platform will help build D&H's next-generation enterprise data platform supporting the modernization of enterprise analytics. This role designs, builds, and operates scalable, production-grade data pipelines across Databricks, SAP data environments, and other cloud and enterprise sources. Candidates are not expected to bring deep expertise in every technology within D&H's target architecture. Successful candidates will combine strong foundational data engineering experience with deep expertise in either modern Databricks/Spark engineering or SAP data engineering, and the ability and interest to develop broader capability across the integrated platform.

Responsibilities:

• Design, develop, test, deploy, and support scalable data pipelines across SAP, Databricks, and related cloud and enterprise environments.
• Develop production-grade ingestion and transformation workloads using SQL and appropriate distributed data-processing technologies.
• Apply reusable engineering patterns for orchestration, data quality, observability, monitoring, error handling, and recovery.
• Collaborate with SAP, Data Platform, Infrastructure, Security, Architecture, Analytics, and business teams to deliver reliable end-to-end data solutions.
• Contribute to lakehouse and medallion architecture patterns and the development of governed, reusable data products.
• Optimize data workloads for performance, scalability, reliability, maintainability, and cost efficiency.
• Follow engineering practices for source control, automated testing, CI/CD, deployment, documentation, and environment promotion.
• Support migration from legacy EDW/ETL platforms to modern cloud-based architectures while validating completeness and data integrity.
• Deliver trusted and certified datasets for enterprise reporting, analytics, planning, and AI use cases.
• Participate in design reviews, troubleshooting, production support, knowledge transfer, and continuous improvement of team standards.
• Develop broader proficiency across D&Hs integrated SAP and Databricks platform through applied project work, mentoring, and training.

First-Year Success:

• Establish and apply production-ready data engineering patterns that support D&Hs modern data platform.
• Deliver reliable data pipelines connecting SAP data environments, Databricks, and other enterprise sources.
• Build capabilities for ingestion, transformation, orchestration, monitoring, recovery, and data quality.
• Contribute specialized depth in either Databricks/Spark engineering or SAP data engineering while developing working knowledge of the broader platform.
• Help transition implementation knowledge, technical patterns, and operational responsibility into D&Hs internal team.

Requirements:

• 8+ years of progressive experience in data engineering, data integration, ETL/ELT, or enterprise data platform development.
• Bachelors degree in Computer Science, Data Engineering, Information Systems, or a related field; or an equivalent combination of education, training, and relevant professional experience.
• Strong hands-on experience with either modern Databricks/Spark data engineering or SAP enterprise data engineering, with the ability to contribute across integrated data environments.
• Advanced SQL skills and strong knowledge of data modeling and scalable data-transformation patterns.
• Experience building and supporting production-grade data pipelines, including orchestration, monitoring, data quality, error handling, recovery, and performance optimization.
• Experience working with cloud or distributed data architectures.
• Ability to translate business and technical requirements into scalable, maintainable data solutions.
• Strong problem-solving, collaboration, documentation, and communication skills across technical and business teams.
• Ability to work effectively in a modernization environment where technologies, standards, and operating practices continue to evolve.

Preferred Qualifications:

Depth in one or more of the following areas is preferred. Expertise across all listed technologies is not required.

Databricks and Modern Data Engineering

• Databricks, Apache Spark, PySpark, Spark SQL, DataFrames, or comparable distributed data-processing technologies.
• Delta Lake, Unity Catalog, Databricks Workflows/Lakeflow, Delta Live Tables/Declarative pipelines, or Databricks SQL].
• Lakehouse and medallion architecture patterns.

SAP Data Engineering

• SAP Datasphere, Graphical/SQL views, Analytical Models, Replication/Transformation Flows, SAP HANA Calculation views and/or Stored Procedures within HDI container.
• SAP BW/4HANA, BW Bridge/BW Model Transfer, Data product generator, HANA Calculation Views.
• SAP S/4HANA data structures, CDS Views, SAP ODP data extraction or integration patterns, ODATA APIs.

Cloud and Engineering Practices

• Microsoft Azure, including cloud storage and supporting security, networking, or identity concepts.
• Git/source control, CI/CD, automated testing, automated deployment, and DataOps/DevOps practices.
• Metadata, lineage, role-based access, data quality, monitoring, and observability.
• Streaming or event-driven technologies such as Apache Kafka or comparable platforms.

Analytics Ecosystem

• Semantic modeling and governed data-product development.
• SAP Analytics Cloud, Power BI, Tableau, Cognos, or comparable enterprise analytics platforms.
• Databricks and/or SAP technical certifications are a plus.

EOE