Big Data Cloud Engineer

Title : Lead Big Data Cloud Engineer

Location : Remote. Candidates must be willing to work onsite in Reston, VA or Washington, DC, once per month for all-hands meetings.

Overview & Responsibilities :

We are seeking a Lead Big Data Cloud Engineer for a crucial production support role with a leading healthcare services client. You will be responsible for the administration and support of the "Enterprise Data Hub" (EDH), a complex Big Data platform built on Cloudera (CDP) and running in the AWS cloud. This is a hands-on role for a lead engineer experienced with the Cloudera ecosystem (including Kafka, Nifi, Hbase, and Spark) who will ensure the health, performance, and security of 15 different data products and real-time data pipelines. You will…

• Provide lead-level production support for the Cloudera (CDP) platform running on AWS.
• Monitor the health of all Hadoop daemon services and Cloudera services using Cloudera Manager.
• Administer and troubleshoot the real-time data pipeline, including Apache Kafka (brokers, topics, streams) and Apache Nifi (flow management, registry).
• Support and troubleshoot other ecosystem components, including Hbase and Solr.
• Write and maintain shell scripts for health checks and monitoring.
• Support data migrations and perform incremental updates / upgrades to the Cloudera environment.
• Manage job workflows (Oozie, Hue) and implement security policies (Ranger).
• Participate in a 24 / 7 on-call rotation to support the production platform.

Required Qualifications :

Preferred Qualifications :