Data Engineer
The Data Engineer is a technical leader and hands-on developer responsible for designing, building, and optimizing data pipelines and infrastructure to support analytics and reporting. This role will serve as the lead developer on strategic data initiatives, ensuring scalable, high-performance solutions are delivered effectively and efficiently.
The ideal candidate is self-directed, thrives in a fast-paced project environment, and is comfortable making technical decisions and architectural recommendations. The ideal candidate has prior experience in modern data platforms, most notable Databricks and the lakehouse architecture. They will work closely with cross-functional teams, including business stakeholders, data analysts, and engineering teams, to develop data solutions that align with enterprise strategies and business goals.
Experience in the financial industry is a plus, particularly in designing secure and compliant data solutions.
Responsibilities :
- Design, build, and maintain scalable ETL / ELT pipelines for structured and unstructured data.
- Optimize data storage, retrieval, and processing for performance, security, and cost-efficiency.
- Ensure data integrity and governance by implementing robust validation, monitoring, and compliance processes.
- Consume and analyze data from the data pipeline to infer, predict and recommend actionable insight, which will inform operational and strategic decision making to produce better results.
- Empower departments and internal consumers with metrics and business intelligence to operate and direct our business, better serving our end customers.
- Determine technical and behavioral requirements, identify strategies as solutions, and section solutions based on resource constraints.
- Work with the business, process owners, and IT team members to design solutions for data and advanced analytics solutions.
- Perform data modeling and prepare data in databases for analysis and reporting through various analytics tools.
- Play a technical specialist role in championing data as a corporate asset.
- Provide technical expertise in collaborating with project and other IT teams, internal and external to the company.
- Contribute to and maintain system data standards.
- Research and recommend innovative, and where possible automated approaches for system data administration tasks. Identify approaches that leverage our resources and provide economies of scale.
- Engineer system that balances and meets performance, scalability, recoverability (including backup design), maintainability, security, high availability requirements and objectives.
Skills :
Databricks and related SQL, Python, PySpark, Delta Live Tables, Data pipelines, AWS S3 object storage, Parquet / Columnar file formats, AWS Glue.Systems Analysis - The application of systems analysis techniques and procedures, including consulting with users, to determine hardware, software, platform, or system functional specifications.Time Management - Managing one's own time and the time of others.Active Listening - Giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.Critical Thinking - Using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions or approaches to problems.Active Learning - Understanding the implications of new information for both current and future problem-solving and decision-making.Writing - Communicating effectively in writing as appropriate for the needs of the audience.Speaking - Talking to others to convey information effectively.Instructing - Teaching others how to do something.Service Orientation - Actively looking for ways to help people.Complex Problem Solving - Identifying complex problems and reviewing related information to develop and evaluate options and implement solutions.Troubleshooting - Determining causes of operating errors and deciding what to do about it.Judgment and Decision Making - Considering the relative costs and benefits of potential actions to choose the most appropriate one.Experience and Education :
High School Diploma (or GED or High School Equivalence Certificate).Associate degree or equivalent training and certification.5+ years of experience in data engineering including SQL, data warehousing, cloud-based data platforms.Databricks experience.2+ years Project Lead or Supervisory experience preferred.Must be legally authorized to work in the United States. We are unable to sponsor or take over sponsorship at this time.