Senior Data Engineer

TentialFort Mill, SC, United States

7 hours ago

Job type

Full-time

Job description

The position is part of a data-focused team working on the "T-Bar" (Transaction Books and Records) product, a critical data store for positions, transactions, and tax lots. The team is transitioning from a legacy data center to a cloud-native AWS environment, aiming to modernize infrastructure and processes by next summer.

Senior Engineer

Must-Have Requirements

Experience with Batch and Streaming Data Processing : Ability to handle intraday use cases with trading partners, including micro-batches.
CI / CD and Developer Discipline : Proficiency in code commit practices, including tying commits to Jira tickets for traceability and cherry-picking.
AWS Familiarity : Experience working within AWS environments, particularly with data pipelines and foundational data layers.
Python : Strong skills in Python for data processing and logic conversion (e.g., rewriting stored procedures from legacy systems).
Infrastructure as Code (IaC) : Experience with Terraform for deploying and managing infrastructure, particularly for carving out separate AWS accounts.

Nice-to-Have Skills

Orchestration Tools : Familiarity with Airflow, Dagster, or similar tools for centralized orchestration (current setup includes step functions, Eventbridge, and an in-house eventing system).

AI / ML Integration : Exposure to AI tools like GitHub Copilot or Cursor for code development, unit test generation, or data quality checks (e.g., anomaly detection).

Data Governance : Experience with data catalogs, producing / consuming assets, or tools like AWS DataZone for governance and contract-based testing.

Event-Driven Architecture : Understanding of event-driven systems, as the team aims to move toward this model in the future.

#DICE

#LI-AM1

Create a job alert for this search

Senior Data Engineer • Fort Mill, SC, United States