Sr Spark Snowflake Engineer (Python)

Job description

JOB: Sr Spark Snowflake Engineer (Python)

Remote

We are seeking a highly experienced Senior Spark Snowflake Data Engineer with 10+ years of expertise in Data Engineering, Enterprise Data Warehousing, Cloud Data Platforms, API Development, and Data Modernization initiatives. The ideal candidate should have extensive experience in Enterprise Data Warehouse (EDW) migration programs, building scalable data pipelines, ingesting and transforming large datasets into Snowflake, and developing enterprise-grade APIs and data services.

The candidate should possess strong hands-on expertise in Python, Apache Spark, Scala, Snowflake, REST APIs, and Cloud Data Engineering, along with a solid understanding of the Healthcare Payer domain. The role demands technical leadership, architectural participation, client interaction, mentoring, work allocation, and delivering highly scalable and reliable data solutions.

Key Responsibilities

Python Development

• Develop high-performance data processing applications using Python.
• Create reusable libraries, frameworks, and utilities.
• Build automation tools supporting migration and operational activities.
• Develop data validation and reconciliation utilities.
• Implement unit testing frameworks and code quality standards.
• Follow secure coding practices and engineering best practices.

Required Experience

• 10+ years of overall Data Engineering experience.
• Strong hands-on experience with Snowflake, Python, Apache Spark, and Scala.
• Experience in large-scale EDW migration programs.
• Experience building enterprise-grade APIs and microservices.
• Experience handling large healthcare datasets.
• Proven experience in solution design, stakeholder interaction, and technical leadership.

Spark & Scala Development

• Develop distributed processing solutions using Apache Spark.
• Build scalable transformations using:
  • Spark SQL
  • PySpark
  • Scala
• Optimize Spark workloads for large-scale healthcare datasets.
• Troubleshoot complex processing and performance bottlenecks.
• Implement partitioning, caching, and workload optimization strategies.
• Support both batch and streaming data processing frameworks.
• Design, develop, and maintain scalable data ingestion, transformation, and processing pipelines.
• Build enterprise-grade ETL/ELT frameworks using Python, Spark, and Snowflake.
• Develop reusable data engineering components, utilities, and automation frameworks.
• Process structured, semi-structured, and unstructured datasets.
• Implement data cleansing, enrichment, validation, and standardization frameworks.
• Optimize data pipelines for performance, scalability, and reliability.
• Support batch and near real-time data processing requirements.

Enterprise Data Warehouse (EDW) Migration & Modernization

• Participate in large-scale data migration initiatives from legacy EDW platforms to Snowflake.
• Analyze source systems, data models, ETL workflows, and reporting structures.
• Support migrations from:
  • Teradata
  • Netezza
  • Oracle
  • SQL Server
  • Hadoop
  • Other Enterprise Data Warehouses
• Develop source-to-target mappings and transformation logic.
• Build migration validation and reconciliation frameworks.
• Ensure high-quality migration with minimal business disruption.

Snowflake Design and implement scalable Snowflake-based data solutions.

• Develop ingestion frameworks into Snowflake.
• Build and optimize:
  • Snowflake Data Pipelines
  • Enterprise Data Warehouses
  • Data Marts
  • Data Sharing Solutions
  • Secure Data Access Layers
• Work with Snowflake capabilities including:
  • Snowpipe
  • Streams & Tasks
  • Time Travel
  • Clustering
  • Secure Views
  • Data Sharing
  • Zero-Copy Cloning
• Optimize Snowflake compute, storage, and query performance.

API Development & Integration

• Design, develop, and maintain RESTful APIs and microservices.
• Build APIs supporting:
  • Data Access Services
  • Data Exchange Solutions
  • Provider Integrations
  • Claims Data Services
  • Analytics Services
• Implement API security, authentication, and authorization controls.
• Manage API lifecycle and versioning strategies.
• Collaborate with application and integration teams for enterprise-wide data consumption.

Healthcare Payer Domain Expertise

Develop and support solutions across:

• Membership & Enrollment
• Claims Processing
• Provider Management
• Care Management
• Regulatory & Quality Reporting
• Pharmacy & Clinical Data

Client & Stakeholder Management

• Interact directly with client business and technology stakeholders.
• Participate in requirement gathering workshops, solution reviews, and design discussions.
• Clarify technical requirements and business needs.
• Provide project status updates and technical guidance.
• Support issue resolution and risk mitigation activities.
• Drive successful delivery of business-critical data initiatives.

API & Integration

• REST APIs
• Microservices
• API Security
• API Lifecycle Management

Databases

• Teradata
• Netezza
• Oracle
• SQL Server

Cloud Technologies

• Cloud Data Platforms
• Data Modernization Solutions
• Cloud Migration Projects

Strong understanding of:

• Claims Processing
• Membership & Enrollment
• Provider Management
• Care Management
• Regulatory Reporting
• Pharmacy Data
• Clinical Data Integration

Preferred Qualifications

• Experience with cloud-based data modernization initiatives.
• Experience leading offshore/onshore delivery teams.
• Snowflake certification preferred.
• Strong communication and client-facing skills.
• Ability to mentor junior engineers and lead technical discussions.