Senior Data Engineer - Spark, AirflowSigmaways Inc • Santa Rosa, CA, United States
Senior Data Engineer - Spark, Airflow
Sigmaways Inc • Santa Rosa, CA, United States
Hace 19 horas
Tipo de contrato
- A tiempo completo
Descripción del trabajoBachelor’s degree in Computer Science, Data Engineering, or related field (or equivalent experience). At least 7 years of experience in data engineering or big data development. Strong expertise in Apache Spark architecture, optimization, and job configuration. Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring. Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems. Expertise in Python programming including data structures and algorithmic problem-solving. Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters. Proficient in shell scripting, including managing and passing variables between scripts. Experienced with spark submit for deployment and tuning. Solid understanding of ETL design, workflow automation, and distributed data systems. Excellent debugging and problem-solving skills in large-scale environments. Experience with AWS Glue, EMR, Databricks, or similar Spark platforms. Knowledge of data lineage and data quality frameworks like Apache Atlas. Familiarity with CI / CD pipelines, Docker / Kubernetes, and data governance tools.
We are seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive our global data and analytics initiatives.
In this role, you will leverage technologies such as Apache Spark , Airflow , and Python to build high performance data processing systems and ensure data quality, reliability, and lineage across Mastercard’s data ecosystem.
The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performance tuning to deliver impactful, data-driven solutions at enterprise scale.
Responsibilities :
- Design and optimize Spark-based ETL pipelines for large-scale data processing.
- Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing.
- Implement partitioning and shuffling strategies to improve Spark performance.
- Ensure data lineage, quality, and traceability across systems.
- Develop Python scripts for data transformation, aggregation, and validation.
- Execute and tune Spark jobs using spark-submit.
- Perform DataFrame joins and aggregations for analytical insights.
- Automate multi-step processes through shell scripting and variable management.
- Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions.
Qualifications :
Crear una alerta de empleo para esta búsqueda
Senior Data Engineer • Santa Rosa, CA, United States
Ofertas relacionadas
AlliedTravelCareers is working with Prime Time Healthcare to find a qualified Cath Lab Tech in San Rafael, California, 94901!.
Now Hiring : Allied Healthcare Cath Lab - San Rafael, CA.Contact us for ...Mostrar más
Farda is reinventing medication adherence with a smart vial and AI-powered platform that automatically tracks ingestion, detects deviations, and syncs real-time data to clinicians and study teams.P...Mostrar más
Travel Radiology : Interventional Radiology Tech San Rafael.Shift Details : 8H Days (8 : 30 AM-5 : 00 PM).Length : 26 WEEKS 26 weeks.
Apply for specific facility Tech.Mostrar más
In this role, you will leverage technologies such as.The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performan...Mostrar más
Senior Backend Engineer – AI Application Builder.SoMa, San Francisco, CA | 💼 Full-Time | Onsite.Salary : $200K–$325K + Meaningful Equity (1%+).
Join an early-stage AI SaaS start-up creating an.AI co...Mostrar más
Homebound is on a mission to make it possible for anyone anywhere to build a home using technology.Created by an experienced team of construction real estate design and technology experts Homebound...Mostrar más
Senior Bioinformatics and Data Scientist.Buck Institute for Research on Aging to work on projects in academia with a close-knit team, collaborate with experts in the field of aging, and participate...Mostrar más
At Visual Concepts, we believe great games are made by diverse and empowered teams with a shared passion for play.As one of the world’s top game development studios, we have shipped over 100 multi-...Mostrar más
We’re looking for a Lead Data Engineer to spearhead the design, implementation, and iteration of a world-class, modern data infrastructure that powers analytics, data science, and ML / AI systems.You...Mostrar más
Krane is the first platform designed to simplify material management across the construction supply chain with an AI-driven end-to-end solution.
With over 85 years of combined experience in construc...Mostrar más
We're the data team behind Midjourney's image generation models.We handle the dataset side : processing, filtering, scoring, captioning, and all the distributed compute that makes high-quality train...Mostrar más Midjourney is a research lab exploring new mediums to expand the imaginative powers of the human species.We are a small, self-funded team focused on design, human infrastructure, and AI.We have no ...Mostrar más 
Full Stack Engineer (Django + React).On-site / Hybrid (San Francisco).DataInsight is a fast-growing, well-funded startup with strong early traction within healthcare.
Our platform is reshaping how c...Mostrar más
Job Description : Join an elite group of sales professionals bringing customized, white glove experiences directly in the customer’s home.
Field Sales Representatives at AT&T are driven to connect – ...Mostrar más
Mercor is hiring experienced Python Engineers • • to support a variety of high-impact research collaborations with leading AI labs.
Freelancers will help improve AI systems through work extending codi...Mostrar más
We are actively searching for a Data Infrastructure Engineer to join our team on a permanent basis.In this founding engineer role you will focus on building next-generation data infrastructure for ...Mostrar más
Director of Evaluation & Learning.Degrees is unlocking student success at scale by giving more students equitable opportunities for a quality college education and career success.The most efficient...Mostrar más
Honey Health is the all-in-one AI back office for primary and specialty care.Our AI agents autonomously handle core back-office jobs, such as aggregating patients data, processing orders and prescr...Mostrar más
Travel Cath Lab Tech - $2,460 to $2,750 per week in San Rafael, CA
AlliedTravelCareers • San Rafael, CA, US
A tiempo completo
Última actualización: hace más de 30 días • Oferta promocionada
Founding Full Stack Engineer
farda. • Santa Rosa, CA, US
A tiempo completo
Última actualización: hace 1 día • Oferta promocionada
Travel Interventional Radiology (IR) - $2,165 per week in San Rafael, CA
Triage Staffing LLC • San Rafael, CA, US
A tiempo completo
Última actualización: hace 21 días • Oferta promocionada
Senior Data Engineer - Spark, Airflow (Santa Rosa)
Sigmaways Inc • Santa Rosa, CA, United States
A tiempo completo
Última actualización: hace 14 horas • Oferta promocionada • Nueva oferta
Senior Back End Engineer - AI Workflow / Application Builder
Ikuto • Santa Rosa, CA, United States
A tiempo completo
Última actualización: hace 13 horas • Oferta promocionada • Nueva oferta
Senior or Staff AI Engineer
Homebound • Santa Rosa, California, USA
A tiempo completo
Última actualización: hace 15 días • Oferta promocionada
Senior Bioinformatics and Data Scientist Furman lab
Buck Institute • Novato, California, United States
A tiempo completo
Última actualización: hace más de 30 días • Oferta promocionada
Senior Backend Engineer, NBA 2K
2k • Novato, California, United States
A tiempo completo
Última actualización: hace más de 30 días • Oferta promocionada
Lead Data Engineer
Mentor Talent Acquisition • Santa Rosa, CA, United States
A tiempo completo
Última actualización: hace 19 horas • Oferta promocionada • Nueva oferta
Full Stack Engineer
Krane • Santa Rosa, CA, US
A tiempo completo
Última actualización: hace menos de 1 hora • Oferta promocionada • Nueva oferta
Senior ML Data Engineer
Midjourney • Santa Rosa, CA, United States
A tiempo completo
Última actualización: hace 19 horas • Oferta promocionada • Nueva oferta
Data Engineer
Midjourney • Santa Rosa, CA, United States
A tiempo completo
Última actualización: hace 19 horas • Oferta promocionada • Nueva oferta
Full Stack Engineer
DataInsight • Santa Rosa, CA, US
A tiempo completo
Última actualización: hace 1 día • Oferta promocionada
Field Sales Representative
AT&T • Point Reyes Station, CA, US
A tiempo completo
Última actualización: hace 2 días • Oferta promocionada
Remote Backend Software Engineer : Python - AI Trainer ($80-$120 per hour)
Mercor • Novato, California, US
Teletrabajo
A tiempo parcial
Última actualización: hace 20 horas • Oferta promocionada • Nueva oferta
Data Platform Engineer / AI Workloads (Santa Rosa)
The Crypto Recruiters • Santa Rosa, CA, US
A tiempo parcial +1
Última actualización: hace 21 horas • Oferta promocionada • Nueva oferta
Data Analyst
10,000 Degrees • San Rafael, California, United States
A tiempo completo
Última actualización: hace 8 días • Oferta promocionada
Senior Software Engineer - AI Agent Infrastructure (Healthcare)
Honey Health • Santa Rosa, CA, US
A tiempo completo
Última actualización: hace 21 días • Oferta promocionada