Senior Lead Azure Streaming Data Engineer

Senior / Lead Azure Streaming Data Engineer . Ideal candidate is senior local resource. Mustb be based in Minneapolis, MN. US Citizen or US green Card holder

The Sr. / Lead Azure Streaming Data Engineer will design and build the real-time data ingestion and processing pipelines that power Client's situational awareness and operational dashboards. This senior contractor will collaborate with the Data Architect and lead on the design of streaming solutions within Azure ensuring that data from APIs, IoT sensors, and MQTT sources is captured, processed, and delivered to end users in seconds. The role balances hands-on engineering of streaming pipelines with solution design and best-practice governance for a secure, scalable Azure environment. You will work closely with the Data Architect, GIS team, and Operations to integrate real-time data feeds (with ArcGIS geospatial context handled by the GIS specialists) and ensure that streaming data is archived and made available in the Lakehouse (Delta Raw / Enriched tables) as well as pushed to live dashboards for immediate insights.

Key Responsibilities :

• Design; Develop Real-Time Pipelines : Design and implement scalable, low- latency data pipelines to ingest real-time data from various sources (REST APIs, MQTT feeds, IoT sensors, etc.) using Azure streaming services such as Azure Event Hubs, Azure IoT Hub, Azure Stream Analytics, and Azure Functions.
• Stream Processing; Delta Lake Integration : Build streaming or micro-batch processing jobs (using Stream Analytics or Synapse Spark Structured Streaming) that cleanse, transform, and write event data into Delta Lake tables on ADLS Gen2 in the Raw and Enriched zones. Ensure that incoming data is captured in the Raw layer and promptly processed into Enriched for use in analytics.
• Real-Time Data Delivery : Enable real-time and near-real-time analytics by integrating streaming outputs with Power BI (GCC) real-time datasets and dashboards. Ensure that critical events (e.g. facility sensor alerts, airline flight alerts, passenger processing kiosk messages) are pushed to Power BI or other subscribers with minimal latency for situational awareness. Coordinate with the GIS team to incorporate location data (ArcGIS) into streaming insights when applicable (the GIS team will handle ArcGIS specifics, but this role will ensure data feeds include necessary spatial references).
• Azure Best Practices; Security : Apply Azure best practices in all solutions - including secure networking (VNet integration, private endpoints for Event Hubs / Storage), authentication / authorization (managed identities, RBAC), encryption, and scalable design (throughput units, partitioning strategies, etc.). Ensure compliance with any relevant security or regulatory standards in this public-sector environment while optimizing for performance.
• Collaboration; Leadership : Work closely with the Client's Data Architect to review designs and align with the overall lakehouse framework. Provide technical leadership on streaming to more junior engineers if needed, and collaborate with stakeholders in operations and IT to understand real-time data needs. Partake in an agile-lite development process (task refinement, sprint planning, demos) and document your solutions for knowledge sharing.

Anticipated Project Start Date : 9 / 29 / 2025

Anticipated End Date : 6 / 30 / 2026

Required Qualifications / Skills :

Preferred Qualifications :