Senior Lead AWS Data Engineer

Overview

Lead the design, implementation, and optimization of ETL workflows using Python and Spark. Build and maintain state-of-the-art data platforms on AWS, contributing to a robust Lakehouse architecture. Serve as a hands-on technical leader, guiding best practices for data engineering solutions and development standards.

Responsibilities

• Lead the design, implementation, and optimization of ETL workflows using Python and Spark.
• Build and maintain state-of-the-art data platforms on AWS, contributing to a robust Lakehouse architecture.
• Serve as a hands-on technical leader, guiding best practices for data engineering solutions and development standards.
• Work closely with architects, Product Owners, and development teams to decompose projects into Epics and plan technical components.
• Drive the migration of existing data workflows to a Lakehouse architecture, utilizing technologies like Iceberg.
• Implement data pipelines using AWS services such as EMR, Glue, Lambda, Step Functions, API Gateway, and Athena.
• Collaborate within an Agile team environment, ensuring alignment and clear communication of complex technical concepts.
• Ensure comprehensive and accessible technical documentation for knowledge sharing and compliance.
• Uphold quality assurance through code reviews, automated testing, and data validation best practices.
• Leverage automation and CI / CD pipelines to streamline development, testing, and deployment processes.
• Remain adaptable to project-specific technology needs, with deep expertise in Python and Spark.
• (Bonus) Apply financial services experience, particularly with equity, fixed income, and index data, if available.
• (Bonus) Utilize solution architecture experience and relevant certifications (e.g., AWS Solutions Architect, Data Analytics).

Seniority level

Employment type

Job function

#J-18808-Ljbffr