Data EngineerEXL • Philadelphia, Pennsylvania, United States

No longer accepting applications

Data Engineer

EXL • Philadelphia, Pennsylvania, United States

9 hours ago

Job type

Full-time

Job description

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

Design, develop, and optimize PySpark jobs for efficient large-scale data processing.

Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.

Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.

Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.

Provide daily coordination and technical oversight to offshore engineering teams.

Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.

Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications :

Primary Skills (Must-Have)

PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.

SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).

Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.

Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.

MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.

DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).

Create a job alert for this search

Data Engineer • Philadelphia, Pennsylvania, United States

Related jobs

Data Architect

Novocure Inc • Philadelphia, PA, United States

Full-time

We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

EXL • Philadelphia, PA, United States

Full-time

Location : Philadelphia, PA (Hybrid).The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements. Data Pipeline Management & S...Show more

Last updated: less than 1 hour ago • Promoted • New!

Lead Databricks Data Engineer

eTeam Inc • Philadelphia, Pennsylvania, United States

Full-time

Quick Apply

We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical leader, you will lever...Show more

Last updated: 30+ days ago

Snowflake dbt Data Migration Engineer (Philadelphia)

Elsevier • Philadelphia, PA, US

Part-time

Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...Show more

Last updated: 11 hours ago • Promoted • New!

Data Scientist / AI Engineer

Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US

Full-time

Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...Show more

Last updated: 30+ days ago

Testing Data Engineer, Remote EST

Open Systems Technologies • Philadelphia, Pennsylvania, United States

Remote

Full-time

Testing Data Engineer Fully Remote, Core hours 9am - 5pm EST.Skill set includes : Expertise on Mainframe Systems Test Management Data Analytics Mainframe Database and Data Structure Skilled in Debug...Show more

Last updated: 1 day ago • Promoted

Drexel Co-Op : Data Engineer

NBME • Philadelphia, PA, USA

Full-time

Quick Apply

NBME is proud to partner with Drexel University and give students the opportunity to utilize the skills they have learned in the classroom and leverage them in an educational setting.Responsible fo...Show more

Last updated: 27 days ago

Cloud Engineer

Meta Force Technology Staffing LLC • Philadelphia, Pennsylvania, USA

Full-time

The Sr Cloud Engineer is responsible for ensuring successful implementation accompanying operational processes support and documentation of cloud and hybrid cloud projects primarily but not limited...Show more

Last updated: 15 days ago • Promoted

Senior Data Engineer (Philadelphia)

Eigen X • Philadelphia, PA, US

Part-time

We are seeking a passionate and skilled Senior Data Engineer to join our dynamic team in Philadelphia, PA.In this role, you will lead the design and implementation of advanced data pipelines for Bu...Show more

Last updated: 1 hour ago • Promoted • New!

Big data / Azure Data Engineer Lead with Healthcare PAYER background

Smart IT Frame LLC • Philadelphia, Pennsylvania, United States

Full-time

Role : Big data lead (With Healthcare PAYER background).Location : Philadelphia, PA (Hybrid – Onsite 3 days / week).At Smart IT Frame, we connect top talent with leading organizations across the USA.Wi...Show more

Last updated: 5 hours ago • Promoted • New!

Data Engineer (Philadelphia)

Realtime Recruitment • Philadelphia, PA, United States

Full-time

Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...Show more

Last updated: 13 hours ago • Promoted • New!

Data Pipeline, ETL Engineer

IntegriChain • Philadelphia, Pennsylvania, USA

Full-time

We are seeking a Data Pipeline / ETL Engineer to design build and maintain the data ingestion and transformation processes that power IntegriChains analytics and SaaS platform.Youll work closely wi...Show more

Last updated: 21 days ago • Promoted

Security Engineer | Data Protection| Experienced Hire

Susquehanna International Group • Philadelphia, PA, United States

Full-time

We are seeking a Security Professional to join our team of engineers who are responsible for understanding and protecting against today's cyber threats in our Bala Cynwyd, PA office.We leverage our...Show more

Last updated: 20 days ago • Promoted

Analytics & Data Engineer

Weidenhammer • Philadelphia, Pennsylvania, United States, 19610

Full-time

The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...Show more

Last updated: 3 days ago

Lead Data Engineer (AWS, Azure, GCP)

Captech Consulting • Philadelphia, Pennsylvania, United States

Full-time

CapTech is an award-winning consulting firm that collaborates with clients to achieve what’s possible through the power of technology. At CapTech, we’re passionate about the work we do and the resul...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

e&e IT Consulting Services, Inc. • Philadelphia, PA, United States

Full-time

Data Engineer for an onsite contract opportunity in Philadelphia, PA!.The Data Engineer is responsible for designing, developing, and optimizing scalable data pipelines and modern data infrastructu...Show more

Last updated: less than 1 hour ago • Promoted • New!

Application Data Platform Engineer (Philadelphia)

MKS2 Technologies • Philadelphia, PA, United States

Full-time +1

Application Data Platform Engineer.Philadelphia, PA - hybrid schedule - 2 days on-site per week.Pay : $150,000 (negotiable) - full benefits. Seeking to hire an Oracle Application Data Platform Engine...Show more

Last updated: 13 hours ago • Promoted • New!

AI / ML Data Engineer

Chabez Tech • Berlin, New Jersey, United States

Temporary

We are seeking a highly skilled and motivated AI Engineer to join our data and analytics team.This role is ideal for someone passionate about building intelligent data-driven solutions using cuttin...Show more

Last updated: 23 days ago • Promoted