Data EngineerEXL • Philadelphia, Pennsylvania, United States

Data Engineer

EXL • Philadelphia, Pennsylvania, United States

23 hours ago

Job type

Full-time

Job description

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

Design, develop, and optimize PySpark jobs for efficient large-scale data processing.

Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.

Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.

Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.

Provide daily coordination and technical oversight to offshore engineering teams.

Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.

Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications :

Primary Skills (Must-Have)

PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.

SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).

Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.

Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.

MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.

DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).

Create a job alert for this search

Data Engineer • Philadelphia, Pennsylvania, United States

Related jobs

Data Architect

Novocure Inc • Philadelphia, PA, United States

Full-time

We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

EXL • Philadelphia, PA, United States

Full-time

Location : Philadelphia, PA (Hybrid).The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements. Data Pipeline Management & S...Show more

Last updated: 14 hours ago • Promoted • New!

Lead Databricks Data Engineer

eTeam Inc • Philadelphia, Pennsylvania, United States

Full-time

Quick Apply

We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical leader, you will lever...Show more

Last updated: 30+ days ago

Snowflake dbt Data Migration Engineer (Philadelphia)

Elsevier • Philadelphia, PA, US

Part-time

Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...Show more

Last updated: 1 day ago • Promoted

Data Scientist / AI Engineer

Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US

Full-time

Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...Show more

Last updated: 30+ days ago

Drexel Co-Op : Data Engineer

NBME • Philadelphia, PA, USA

Full-time

Quick Apply

NBME is proud to partner with Drexel University and give students the opportunity to utilize the skills they have learned in the classroom and leverage them in an educational setting.Responsible fo...Show more

Last updated: 28 days ago

Head of Data Science and Advanced Analytics (Philadelphia)

Confidential • Philadelphia, PA, United States

Full-time

Head of Data Science and Advanced Analytics.Globally renowned biopharmaceutical company.The Company is in search of a Head of Data Science and Advanced Analytics to spearhead the strategic and scie...Show more

Last updated: 1 day ago • Promoted

Big data / Azure Data Engineer Lead with Healthcare PAYER background

Smart IT Frame LLC • Philadelphia, PA, United States

Full-time

Role : Big data lead (With Healthcare PAYER background).Location : Philadelphia, PA (Hybrid – Onsite 3 days / week).At Smart IT Frame, we connect top talent with leading organizations across the USA.Wi...Show more

Last updated: 1 day ago • Promoted

Engineer

Quality Talent Group • Lower Makefield, Pennsylvania, United States

Full-time

Quick Apply

Our client is a leading force in advancing safer, smarter AI technology.Their work has been featured in.They’ve built a global community of expert contributors and have already paid out more ...Show more

Last updated: 8 days ago

Data Engineer (Philadelphia)

Realtime Recruitment • Philadelphia, PA, United States

Full-time

Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...Show more

Last updated: 1 day ago • Promoted

Security Engineer | Data Protection| Experienced Hire

Susquehanna International Group • Philadelphia, PA, United States

Full-time

We are seeking a Security Professional to join our team of engineers who are responsible for understanding and protecting against today's cyber threats in our Bala Cynwyd, PA office.We leverage our...Show more

Last updated: 21 days ago • Promoted

Lead Software Engineer

Relativity • Philadelphia, PA, United States

Full-time

We are seeking a Lead Software Engineer to join the Retrieval Ingestion Team at Relativity.This role is ideal for an experienced engineer who thrives on designing and operating high throughput inge...Show more

Last updated: 30+ days ago • Promoted

Analytics & Data Engineer

Weidenhammer • Philadelphia, Pennsylvania, United States, 19610

Full-time

The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...Show more

Last updated: 3 days ago

Senior Data Engineer (Philadelphia)

Eigen X • Philadelphia, PA, United States

Full-time

We are seeking a passionate and skilled Senior Data Engineer to join our dynamic team in Philadelphia, PA.In this role, you will lead the design and implementation of advanced data pipelines for Bu...Show more

Last updated: 12 hours ago • Promoted • New!

Data Engineer

e&e IT Consulting Services, Inc. • Philadelphia, PA, United States

Full-time

Data Engineer for an onsite contract opportunity in Philadelphia, PA!.The Data Engineer is responsible for designing, developing, and optimizing scalable data pipelines and modern data infrastructu...Show more

Last updated: 14 hours ago • Promoted • New!

Time-Series Data Engineer (Doylestown)

Kane Partners LLC • Doylestown, PA, US

Part-time

Local Candidates Only No Sponsorship • •.A growing technology company in the Warrington, PA area is seeking a Data Engineer to join its analytics and machine learning team.This is a hands-on, engine...Show more

Last updated: 1 day ago • Promoted

Application Data Platform Engineer (Philadelphia)

MKS2 Technologies • Philadelphia, PA, United States

Full-time +1

Application Data Platform Engineer.Philadelphia, PA - hybrid schedule - 2 days on-site per week.Pay : $150,000 (negotiable) - full benefits. Seeking to hire an Oracle Application Data Platform Engine...Show more

Last updated: 1 day ago • Promoted

AI / ML Data Engineer

Chabez Tech • Berlin, New Jersey, United States

Temporary

We are seeking a highly skilled and motivated AI Engineer to join our data and analytics team.This role is ideal for someone passionate about building intelligent data-driven solutions using cuttin...Show more

Last updated: 23 days ago • Promoted