Lead PySpark EngineerThecentermemphis • Orlando, FL, US

Lead PySpark Engineer

Thecentermemphis • Orlando, FL, US

14 hours ago

Job type

Full-time

Job description

Overview

This position will follow our hybrid work model, and we expect the selected candidate to be in office 2-3 days a week (10-12 days a month) at the following office location : St. Petersburg, FL.

Job Summary : This Lead Engineer, Technology will partner closely with the Compliance and Supervision Business Unit on solutions from inception through ongoing maintenance. The role will focus on building out Supervision and Compliance Surveillance controls within Wealth Management. The ideal candidate will have extensive experience with AWS Glue, PySpark, and Python programming . This position will be responsible for designing, implementing, and developing business controls and collaborating with cross-functional teams to meet business needs. This hybrid position will be based out of our corporate headquarters in Saint Petersburg, FL.

Responsibilities

Design and develop business controls using AWS Glue and PySpark.
Integrate data from various sources into Amazon Redshift , ensuring data quality and consistency.
Optimize the performance of data processing jobs and fine tune queries.
Monitor and troubleshoot data pipelines to ensure reliability and efficiency.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
Ensure compliance with security policies and regulations.
Document data workflows and best practices.
Mentor and guide junior engineers on the team.
Continuously explore new AWS services and technologies to improve existing solutions.

Skills

Data Engineering Tools :

Experience in Airflow

PySpark and experience with caching, RDD, multithreading, in-memory computations of very large datasets.

Experience in open table formats; e.g., Iceberg, Delta Lake, Avro, Parquet

Solid knowledge and experience with Glue Notebook and EMR / Databricks.

Experience with DataLake concept on distributed object store.

Python programming

Exp SQL : Advanced experience in writing complex SQL queries

Data Warehousing :

Solid knowledge of OLTP and OLAP

AWS Services Experience minimum 3 years : Glue, EMR, S3, Redshift, Athena, Lambda, Step Functions, Data pipeline and migration, Terraform, CI / CD pipeline

Generic domain background :

Data Analysis and Reporting : Proficiency in SQL

System Integration : Experience with integrating compliance systems with other enterprise systems.

Financial Industry Experience : Experience in financial firms and understanding their regulatory environments.

Knowledge in TFS is a plus.

Knowledge in Jenkins pipeline and deployment.

Ability to :

Identify and understand issues, problems and opportunities for automation.

Communicate effectively and organize information to diverse audiences.

Develop solutions aligned with available facts, constraints, and probable consequences.

Maintain a satisfactory level of technical and professional skill; stay current with developments in the field.

Develop and use collaborative relationships to achieve work goals.

Prioritize internal and external client needs and sustain productive client relationships.

Education

Bachelor's : Computer and Information Science (Required)

Bachelor's : Computer Engineering

Possible other related degrees as applicable

Work Experience

General Experience : 6 to 10 years

EEO and Company Information

This position adheres to our equal opportunity employment policy. The company makes employment decisions based on merit and business needs.

Note : This formatted description is intended to reflect the original job information while removing irrelevant boilerplate and ensuring proper HTML structure with allowed tags.

J-18808-Ljbffr

Create a job alert for this search

Lead Engineer • Orlando, FL, US