Data Engineer

Job Description

• Design and deploy scalable ETL / ELT pipelines to ingest, transform, and load clinical data from diverse sources (EMRs,  labs, IoT devices, data lakes, FHIR / HL7 APIs), including Azure and Snowflake.
• Architect and optimize Microsoft Azure and Snowflake environments for clinical data storage, extraction, transformation, and loading, machine learning operations (MLOps) performance tuning, cost management, and secure data sharing.
• Ensure compliance with healthcare regulations (HIPAA, GDPR) by implementing data anonymization, encryption, and audit trails.
• Collaborate with clinical stakeholders to translate business requirements into technical solutions for analytics and reporting.
• Develop and maintain data governance frameworks , including metadata management, data lineage, and quality checks (e.g., validation of lab results, patient demographics).
• Automate data pipelines using orchestration tools (e.g., Apache Airflow, Prefect) and integrate real-time streaming solutions (e.g., Kafka) where applicable.
• Build and maintain documentation for data models, pipelines, and processes to ensure reproducibility and transparency.
• Advanced proficiency in Snowflake (Snowpipe, Time Travel, Zero-Copy Cloning) and SQL for complex transformations.
• Hands-on experience with ETL / ELT tools (Apache Spark, AWS Glue, Azure Data Factory) and cloud platforms (AWS, Azure, GCP).
• Strong programming skills in Python / Scala (Pandas, PySpark) for data scripting and automation.
• Familiarity with healthcare data formats (OMOP, FHIR, HL7, DICOM) and clinical workflows.
• Expertise in federated learning and running large jobs on high-performance computing servers is a plus.
• Data Governance : Ability to implement data quality frameworks (e.g., Great Expectations) and metadata management tools.
• Regulatory Compliance : Proven experience securing PHI / PII data and adhering to HIPAA / GDPR requirements.
• Problem-Solving : Ability to troubleshoot pipeline failures, optimize query performance, and resolve data discrepancies.
• Collaboration : Strong communication skills to work with cross-functional teams (clinicians, analysts, IT).

Requirements

Requirements

Data Engineer with Pharma / Healthcare background