Role : Senior Big Data & DevOps Engineer.
Location : Pittsburgh, PA (Remote).
Duration : Long Term Contract.
Position Summary :
- Seeking a Senior Big Data & DevOps Engineer with 8+ years of expertise in HDFS, Hive, Impala, PySpark, Python, and DevOps automation (uDeploy, Jenkins). Responsible for end-to-end management of enterprise data platforms, including HDFS table governance, scalable ETL pipeline development, CI / CD automation, platform upgrades, incident / change management, and production support.
Key Responsibilities :
Design, manage, and optimize HDFS directories, tables, and data partitioning strategies across large distributed clusters.Enforce data retention, lifecycle policies, and security compliance in high-volume Hadoop environments.Administer Hive and Impala platforms, focusing on performance tuning, capacity planning, and troubleshooting.Develop scalable ETL pipelines and reusable data frameworks using PySpark, Hive, and Python.Implement and support CI / CD pipelines via Jenkins and uDeploy, ensuring automated, reliable, multi-environment deployments.Manage platform upgrades, patching, and code promotion across DEV, QA, UAT, and PROD stages.Perform advanced Linux administration, including scripting, system tuning, disk management, and disaster recovery.Drive incident and change management using ServiceNow, including RCA investigation, issue resolution, and preventive actions.Mentor junior engineers, foster technical leadership, and document operational knowledge for internal sharing.Required Qualifications :
Bachelor's degree in Computer Science or related field.8+ years in Big Data engineering and DevOps.Advanced proficiency in HDFS, Hive, Impala, PySpark, Python, and Linux.Proven CI / CD experience with Jenkins and uDeploy.Strong ETL development, orchestration, and optimization skills.Experience with ServiceNow for incident / change / problem management.Nice to Have :
Experience with cloud-based data platforms (AWS EMR).Knowledge of Docker, Kubernetes, and infrastructure automation (Ansible, Terraform).