Data EngineerEXL • Philadelphia, Pennsylvania, United States

No longer accepting applications

Data Engineer

EXL • Philadelphia, Pennsylvania, United States

6 hours ago

Job type

Full-time

Job description

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

Design, develop, and optimize PySpark jobs for efficient large-scale data processing.

Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.

Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.

Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.

Provide daily coordination and technical oversight to offshore engineering teams.

Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.

Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications :

Primary Skills (Must-Have)

PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.

SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).

Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.

Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.

MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.

DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).

Create a job alert for this search

Data Engineer • Philadelphia, Pennsylvania, United States

Related jobs

Data Architect

Novocure Inc • Philadelphia, PA, United States

Full-time

We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...Show more

Last updated: 30+ days ago • Promoted

Lead Databricks Data Engineer

eTeam Inc • Philadelphia, Pennsylvania, United States

Full-time

Quick Apply

We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical leader, you will lever...Show more

Last updated: 30+ days ago

Data Scientist / AI Engineer

Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US

Full-time

Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...Show more

Last updated: 30+ days ago

Snowflake dbt Data Migration Engineer (Philadelphia)

Elsevier • Philadelphia, PA, United States

Full-time

Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...Show more

Last updated: 7 hours ago • Promoted • New!

Salesforce Developer

Stefanini North America and APAC • Yardley, PA, US

Full-time

We are seeking a highly skilled Salesforce Engineer to design, build, and optimize Salesforce solutions that empower our sales, commercial operations, and marketing teams.The ideal candidate will c...Show more

Last updated: 20 days ago • Promoted

Drexel Co-Op : Data Engineer

NBME • Philadelphia, PA, USA

Full-time

Quick Apply

NBME is proud to partner with Drexel University and give students the opportunity to utilize the skills they have learned in the classroom and leverage them in an educational setting.Responsible fo...Show more

Last updated: 27 days ago

Head of Data Science and Advanced Analytics (Philadelphia)

Confidential • Philadelphia, PA, United States

Full-time

Head of Data Science and Advanced Analytics.Globally renowned biopharmaceutical company.The Company is in search of a Head of Data Science and Advanced Analytics to spearhead the strategic and scie...Show more

Last updated: 7 hours ago • Promoted • New!

Analyst

Tata Consultancy Services • Philadelphia, PA, United States

Full-time

Data Analyst Must Have Technical / Functional Skills 8+ years of experience Experienced Data Analyst with expertise in collecting, cleaning, transforming, and analyzing large datasets to drive busine...Show more

Last updated: 25 days ago • Promoted

Big data / Azure Data Engineer Lead with Healthcare PAYER background

Smart IT Frame LLC • Philadelphia, PA, United States

Full-time

Role : Big data lead (With Healthcare PAYER background).Location : Philadelphia, PA (Hybrid – Onsite 3 days / week).At Smart IT Frame, we connect top talent with leading organizations across the USA.Wi...Show more

Last updated: 9 hours ago • Promoted • New!

Data Engineer (Philadelphia)

Realtime Recruitment • Philadelphia, PA, United States

Full-time

Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...Show more

Last updated: 9 hours ago • Promoted • New!

Security Engineer | Data Protection| Experienced Hire

Susquehanna International Group • Philadelphia, PA, United States

Full-time

We are seeking a Security Professional to join our team of engineers who are responsible for understanding and protecting against today's cyber threats in our Bala Cynwyd, PA office.We leverage our...Show more

Last updated: 20 days ago • Promoted

Lead Software Engineer

Relativity • Philadelphia, PA, United States

Full-time

We are seeking a Lead Software Engineer to join the Retrieval Ingestion Team at Relativity.This role is ideal for an experienced engineer who thrives on designing and operating high throughput inge...Show more

Last updated: 30+ days ago • Promoted

Analytics & Data Engineer

Weidenhammer • Philadelphia, Pennsylvania, United States, 19610

Full-time

The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...Show more

Last updated: 3 days ago

Solutions Engineer (Philadelphia)

Oscar • Philadelphia, PA, US

Full-time +2

Oscar is working with a leading data transformation organization that is looking for an experienced.Act as the primary technical resource for the sales organization, providing support throughout th...Show more

Last updated: 8 hours ago • Promoted • New!

Distribution Engineer

Leidos Inc • Philadelphia, PA, United States

Full-time

We challenge our staff with interesting assignments that provide them with an opportunity to thrive, professionally and personally. For us, helping you grow your career is good business.We're lookin...Show more

Last updated: 30+ days ago • Promoted

Application Data Platform Engineer (Philadelphia)

MKS2 Technologies • Philadelphia, PA, United States

Full-time +1

Application Data Platform Engineer.Philadelphia, PA - hybrid schedule - 2 days on-site per week.Pay : $150,000 (negotiable) - full benefits. Seeking to hire an Oracle Application Data Platform Engine...Show more

Last updated: 9 hours ago • Promoted • New!

Remote Side Hustle Evaluator - Flexible Online Gig Work

Finance Buzz • Glassboro, New Jersey, US

Remote

Temporary

Are you looking to earn extra income from the comfort of your home? We're seeking motivated individuals to explore and test a variety of remote side hustle opportunities featured on FinanceBuzz.Thi...Show more

Last updated: 30+ days ago • Promoted

Time-Series Data Engineer (Doylestown)

Kane Partners LLC • Doylestown, PA, United States

Full-time

Local Candidates Only No Sponsorship • •.A growing technology company in the Warrington, PA area is seeking a Data Engineer to join its analytics and machine learning team.This is a hands-on, engine...Show more

Last updated: 9 hours ago • Promoted • New!