Lead Data Engineer

Insight Global is seeking a Lead Data Engineer for a contract to hire opportunity at one of our trading and commodities clients based in Chicago, IL. This role ideally sits local to the Chicago, IL area, but if a candidate is available to relocate to Chicago, IL that would be considered.

This individual will be responsible for assisting a team in building out the entirety of the data product lifecycle within the modernization and migration of the clients databricks platform and azure cloud. The candidate will be responsible for the build out, maintain scalable data pipelines and cloud (Azure) based infrastructure . Additionally, this candidate will be responsible for ensuring reliable data flow, optimize database performance and support key and critical data driven decision making within the organization. Ability to write efficient, reusable, and testable python scripting / code for data ingestion, transformation, and validation will be a critical function of this role.

Other responsibilities will include :

• Proven experience with Databricks and Azure cloud services (e.g., Data Lake, Synapse, Data Factory).
• Strong proficiency in Python for data engineering tasks
• Solid understanding of ETL / ELT processes, data modeling, and distributed computing
• Experience with SQL and working with large-scale relational and non-relational databases
• Familiarity with CI / CD pipelines and version control - Github for integration

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity / affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and / or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy :

Required Skills & Experience

3-5 years of experience as a Data Engineer migrating from on Prem to Databricks

Some experience with architecture or design in Databricks

Experience with Python and Azure

3-5 years of experience working within the Data Product Lifecycle

3+ years of experience in migrating legacy systems to SQL

Ability to work in a fast paced, highly visible environment

Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and / or other paid time off as provided by applicable law.