Senior Lead Data Engineer
Primary Skills
- AKS, Event Hub, Azure DevOps, Cosmos DB, Azure Functions
Specialization
Azure Data Engineering Advanced : Senior Data EngineerJob requirements
Key Responsibilities :Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQLManage and optimize the infrastructure ensuring reliability and performanceDesign and implement data models and schemas for enterprise data warehouse and data martsWrite and maintain scripts in Python for various applicationsDevelop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databasesWork closely with analysts and product managers to understand data requirements and deliver high-quality dataProvide technical guidance and mentorship to other team members fostering best practices in data engineeringMonitor system performance, identify bottlenecks, implement solutions to ensure high data availabilityRequired Skills :
High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processingGood knowledge in Python for data manipulation and scriptingMastery of advanced SQL and deep experience working with relational databases and datawarehousesHands-on experience with workflow management tools like Apache AirflowFamiliarity with Unix shell scripting for automation and system managementWe may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.