Azure Data Engineer

As an Azure Data Engineer, you will leverage your expertise in Shell scripting, SQL, Spark, Python, Azure AD, and Azure Data Lake Store to drive data-driven insights and solutions. You will work in a hybrid model, collaborating with cross-functional teams to enhance geospatial data analysis and contribute to impactful projects that align with our companys goals. Your role will be pivotal in optimizing data processes and ensuring data integrity.

Experience : 6 to 10 Yrs

Required Skills : Shell scripting, Python, Azure AD, Spark, SQL, Azure Data Lake Store

Responsibilities : We are looking for a highly skilled Platform Staff Engineer to lead the development and maintenance of our Databricks platform, supporting our healthcare delivery initiatives. You will be a hands-on leader, responsible for the architecture, implementation, and operational excellence of our data platform. You will be responsible for ensuring the platforms reliability, performance, and security, while also driving its strategic evolution to meet future business needs. This position requires a blend of deep technical expertise, a proactive mindset, and a commitment to operational excellence. This role sits at the intersection of platform engineering, cloud engineering, data governance, and developer enablement, requiring a blend of deep technical expertise, strategic thinking, and a passion for building scalable, reliable, and secure systems. You will be a trusted partner for our data engineers ensuring they have the tools and best practices to drive business value. Responsibilities Operational Excellence Oversee the operational health of the platform. Maintain and enhance monitoring, logging, and alerting solutions to proactively identify and resolve issues. Lead incident response and perform root cause analysis for platform-wide problems. Platform Architecture & Strategy Maintain and enhance the core components of our Azure Databricks environment. Define the strategic roadmap for the platform, including the adoption of new features, services, and best practices (e.g., Delta Lake, Unity Catalog, Databricks Workflows. Github CI / CD). Security & Governance Manage the enforcement and enhancement of data security and governance policies. Manage the Unity Catalog to ensure fine-grained access control, data lineage, and compliance with internal and external regulations. Performance & Cost Optimization Implement strategies for cost management and performance tuning at a platform level. Provide guidance and tooling to optimize Databricks clusters and workloads to maximize efficiency and minimize cloud spend. Tooling & Automation Maintain and enhance automated solutions for provisioning, scaling, and managing Databricks workspaces and resources using Infrastructure as Code (Terraform) and CI / CD pipelines (GitHub). Developer Enablement Serve as a subject matter expert and mentor to other data engineers and Azure databricks platform engineers. Develop and advocate for best practices, create reusable patterns and libraries, and provide expert-level technical support and training to data teams. Cross-Functional Collaboration Partner with data engineering and business teams to understand their needs and maintain and enhance a platform that empowers them. Work with security, networking, and other infrastructure teams to ensure a cohesive and secure environment. Leading the Platform Engineers Overseeing and directing the efforts of 2 platform engineers.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.