Devops - Airflow

Role - Devops

Location : - Tempe, AZ (Onsite)

Experience - 8+ Years

Job Description

• Airflow Expertise Possessing deep knowledge of Airflows architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.
• Workflow Design and Development Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.
• Performance Optimization Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling.
• Integration with Cloud Services Integrating Airflow with cloud-native services like Azure Data Factory, Azure Databricks, Azure Storage, and others.
• CICD Pipeline Management Developing and maintaining CICD pipelines for Airflow DAG deployment, testing, and version control, often using tools like Azure DevOps.
• Monitoring and Alerting Implementing monitoring, alerting, and logging standards for Airflow jobs to ensure operational excellence and rapid incident response.
• Documentation and Knowledge Sharing Creating and maintaining documentation for Airflow configurations, deployment processes, and operational procedures.
• Mentoring other engineers and leading knowledge-sharing sessions.
• Troubleshooting and Incident Response Providing expert-level troubleshooting support for Airflow-related issues and contributing to incident response efforts.
• Additional Responsibilities Cloud Platform Expertise May require strong knowledge of cloud platforms like Azure (including services like Azure Synapse, Azure Functions, and Azure API Management) or AWS.
• Containerization and Orchestration Experience with Docker and Kubernetes for deploying and managing Airflow in containerized environments. Programming Skills
• Proficiency in Python Java for developing Airflow DAGs and other related components

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.