Senior Spark Developer (Python, AWS, SQL)

Overview

We are seeking a highly skilled Spark Developer with strong experience in Python, AWS, and SQL to join our team. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions, ensuring data quality, scalability, and performance. This role requires a solid background in distributed computing, cloud environments, and data engineering best practices.

Compensation for NYC : 120-150000 USD Gross per year and based on interview results.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark (batch and / or streaming).
• Build, optimize, and manage ETL / ELT workflows integrating multiple data sources.
• Develop data solutions in Python for data transformations, automation, and orchestration.
• Leverage AWS services (S3, EMR, Glue, Lambda, Redshift, Kinesis, etc.) to implement cloud-native data platforms.
• Write efficient SQL queries for data extraction, transformation, and reporting.
• Ensure data quality, lineage, and governance across pipelines.
• Collaborate with data engineers, architects, and analysts to deliver end-to-end data solutions.
• Troubleshoot performance bottlenecks and optimize Spark jobs for speed and cost-efficiency.

Skills

Must have

Nice to have

#J-18808-Ljbffr