Cloud Engineer

Cloud Data Engineer

Location : Available in an AWS Availability Zone (AZ) and able to commute to the office with a few hours' notice. Expected to work from the office regularly during stakeholder or team visits.

Job Summary :

We are seeking a skilled and experienced Cloud Data Engineer to join our dynamic team. You will be responsible for designing, building, and maintaining our cloud-based data platform on AWS, with a strong focus on leveraging EKS for multi-tenant environments and utilizing EMR for big data processing. Your expertise in integrating security through Okta AD groups and Kerberos, along with configuring Jupyter for various computational needs, will be crucial in enabling our data science and analytics teams.

Responsibilities :

Design, implement, and manage multi-tenant EKS clusters to support various data workloads.

Develop and maintain scalable and cost-effective big data processing pipelines using EMR.

Implement and manage security protocols, including integration with Okta AD groups for authentication and authorization, and Kerberos for secure access to data services.

Configure and optimize Jupyter environments to support diverse computational requirements for data scientists and analysts.

Collaborate with data scientists, analysts, and other stakeholders to understand their data and computational needs.

Ensure the reliability, performance, and security of the data platform.

Troubleshoot and resolve issues related to the data platform and its components.

Stay up-to-date with the latest AWS services and best practices in cloud data engineering.

Be available to come to the office with a few hours' notice and work regularly from the office during stakeholder or team visits.

Qualifications :

Proven experience in designing, building, and managing data platforms on AWS.

Strong hands-on experience with Amazon EKS (Elastic Kubernetes Service), including managing multi-tenant environments.

Significant experience with Amazon EMR (Elastic MapReduce) for big data processing.

Experience integrating AWS services with Okta for authentication and authorization.

Solid understanding and practical experience implementing Kerberos for secure data access.

Proficiency in configuring and managing Jupyter environments (e.g., Jupyter Notebook, JupyterLab).

Experience with infrastructure-as-code tools (e.g., Terraform, CloudFormation) is a plus.

Strong understanding of data warehousing concepts and ETL / ELT processes.

Excellent problem-solving and communication skills.

Ability to work independently and collaboratively within a team.

Must be located within a reasonable distance of the office and able to commute with a few hours' notice.

Willingness to work from the office regularly during stakeholder or team visits.