Talent.com
Senior Software Engineer, Dataplane Management

Senior Software Engineer, Dataplane Management

AstronomerSan Francisco, CA, United States
30+ days ago
Job type
  • Full-time
Job description

Astronomer empowers data teams to bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than 700 of the world's leading enterprises, Astronomer lets businesses do more with their data. To learn more, visit www.astronomer.io.

About this role :

At Astronomer, we’re redefining how companies run Apache Airflow at scale. Our R&D organization is home to some of the most innovative minds in cloud infrastructure and open-source software.

We’re looking for a Senior Software Engineer to join our Data Plane Management team, whose mission is to efficiently operate, and develop new features for, the cloud compute foundation that our customer Airflow deployments are hosted upon.

This is a particularly fascinating role for someone who loves the challenge of managing a large scale fleet of 500+ Kubernetes clusters with a true software engineering mindset. This team owns the compute & network primitives in our Astro platform and develops the APIs that our customers use to stand up their environments. The team’s roadmap includes new self-service features such as failover cluster support, VPC / VNet peering, Transit Gateway wiring, metric / log shipping, and more.

What you get to do :

  • Own key endpoints and features of our flagship product, Astro, to extend our offering to more complex customer networking options.
  • Work across domains to develop diverse features for our core infrastructure (ie : workload identity, multitenancy, cross region disaster recovery, cloud resource management, private network access).
  • Evolve our fleet orchestration system to enable us to safely make (and roll back) changes across our infrastructure, and to enable us to scale from the 500+ clusters we have now to thousands in the future.
  • Gain experience working in a multi-cloud environment by working with managed Kubernetes offerings and network / auth primitives from AWS, Azure, and GCP.
  • Deepen your operational knowledge of Kubernetes-based workloads by managing the data pipelines of many of the largest companies in the world.

What you bring to the role :

  • Experience building and operating SaaS infrastructure, or experience managing a large scale internal compute platform.
  • Software engineering expertise with Golang, or similar languages with a desire to learn Golang.
  • A passion for reliability and operational excellence. A low tolerance for toil, alert fatigue, and other nonsense.
  • Production experience with a container based orchestration system (Kubernetes preferred, but not essential).
  • Understanding of how to build with security and isolation in mind, so that Astronomer’s managed platform can securely integrate with any customer environment and ensure strict isolation between customer workloads.
  • Strong communication skills, both written and verbal, with experience in working with a globally distributed team in delivery.
  • Proactive approach to identifying and addressing issues, with a focus on ownership and accountability.
  • Experience as part of an on-call rotation - this role involves periodic on-call for the services and system we own.
  • Bonus points if you have :

  • Experience working on a SaaS / PaaS product across multiple cloud providers, with a ‘single tenant data plane’ model.
  • Programmatically administered Kubernetes in multiple clouds.
  • Experience designing systems for resiliency, scale and security. A familiarity with Non-Abstract Systems design is a major plus.
  • A passion for finding and addressing inefficiencies in code, infrastructure costs, tooling and processes.
  • Experience with our particular tech stack components and technologies (deep breath) : Calico / Azure CNI / Cilium, PostgreSQL / Aurora / CloudSQL / etc, OpenTelemetry, Prometheus / Chronosphere, Splunk, Istio, Karpenter, Falco,
  • Experience of the innards and quirks of AWS, GCP and Azure.
  • Experience with Apache Airflow.
  • The estimated salary for this role ranges from $180,000 - $210,000 based on leveling and geography, along with an equity component and a comprehensive benefits package. This range is merely an estimate; actual compensation may deviate from this range based on skills, experience, and qualifications.

    #LI-Remote

    At Astronomer, we value diversity. We are an equal opportunity employer : we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

    #J-18808-Ljbffr

    Create a job alert for this search

    Senior Software Engineer • San Francisco, CA, United States

    Related jobs
    • Promoted
    Senior Software Engineer, ML Data Platform

    Senior Software Engineer, ML Data Platform

    Dyna RoboticsRedwood City, CA, United States
    Full-time
    Dyna Robotics makes general-purpose robots powered by a proprietary embodied AI foundation model that generalizes and self-improves across varied environments with commercial-grade performance.Dyna...Show moreLast updated: 27 days ago
    • Promoted
    Senior Software Engineer, Data Solutions & Initiatives

    Senior Software Engineer, Data Solutions & Initiatives

    Apple Inc.Cupertino, CA, United States
    Full-time
    Senior Software Engineer, Data Solutions & Initiatives.Cupertino, California, United States Sales and Business Development. Imagine what you could do here.The people here at Apple don’t just create ...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer - Data Transparency

    Senior Software Engineer - Data Transparency

    The Trade DeskSan Jose, CA, United States
    Full-time
    The Trade Desk is a global technology company with a mission to create a better, more open internet for everyone through principled, intelligent advertising. Handling over 1 trillion queries per day...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer, Data

    Senior Software Engineer, Data

    P2PSan Francisco, CA, United States
    Full-time
    Our mission is to bring web3 to a billion people, by providing builders with the tools they need to build exceptional onchain products. Alchemy is the only complete developer platform that offers th...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer, Data Infrastructure (RDBMS)

    Senior Software Engineer, Data Infrastructure (RDBMS)

    TRM LabsSan Francisco, CA, United States
    Full-time
    Senior or Staff Software Engineer, Database Engineer.Senior or Staff Software Engineer, Database Engineer.TRM Labs is a blockchain intelligence company committed to fighting crime and creating a sa...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer - Data

    Senior Software Engineer - Data

    Windfall Data, Inc.San Francisco, CA, United States
    Full-time
    Windfall is seeking a Senior Software Engineer to join our data team.In this role, you will be building out the core data asset that everything else at Windfall is built on top of.Communication and...Show moreLast updated: 2 days ago
    • Promoted
    Senior Software Engineer- Database Infrastructure

    Senior Software Engineer- Database Infrastructure

    King River Capital GroupSan Francisco, CA, United States
    Full-time
    Discord is used by over 200 million people every month for many different reasons, but there’s one thing that nearly everyone does on our platform : . Over 90% of our users play games, spending a comb...Show moreLast updated: 3 days ago
    • Promoted
    Senior Software Engineer, Data Localization

    Senior Software Engineer, Data Localization

    Cloudflare, Inc.San Francisco, CA, United States
    Full-time
    At Cloudflare, we are on a mission to help build a better Internet.Today the company runs one of the world's largest networks that powers millions of websites and other Internet properties for cust...Show moreLast updated: 3 days ago
    • Promoted
    Senior Software Engineer, Data Platform

    Senior Software Engineer, Data Platform

    King River Capital GroupSan Francisco, CA, United States
    Full-time
    Discord is used by over 200 million people every month for many different reasons, but there’s one thing that nearly everyone does on our platform : . Over 90% of our users play games, spending a comb...Show moreLast updated: 25 days ago
    • Promoted
    • New!
    Senior Software Engineer – Cloud Data Platform

    Senior Software Engineer – Cloud Data Platform

    Disneyland Hong KongSan Francisco, CA, United States
    Full-time
    A global entertainment company seeks a Senior Software Engineer to drive data platform innovation.The role involves developing critical tools for engineering teams, ensuring operational excellence,...Show moreLast updated: 9 hours ago
    • Promoted
    Senior Software Engineer - Data Infrastructure

    Senior Software Engineer - Data Infrastructure

    PlaidSan Francisco, CA, United States
    Full-time
    Senior Software Engineer - Data Infrastructure.Making data driven decisions is key to Plaid's culture.To support that, we need to scale our data systems while maintaining correct and complete data....Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer, Data Platform

    Senior Software Engineer, Data Platform

    SkydioSan Mateo, CA, United States
    Full-time
    Skydio is the leading US drone company and the world leader in autonomous flight, the key technology for the future of drones and aerial mobility. The Skydio team combines deep expertise in artifici...Show moreLast updated: 4 days ago
    • Promoted
    Senior Software Engineer, Data Platform

    Senior Software Engineer, Data Platform

    Parafin Inc.San Francisco, CA, United States
    Full-time
    At Parafin, we’re on a mission to grow small businesses.Small businesses are the backbone of our economy, but traditional banks often don’t have their backs. We build tech that makes it simple for s...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer, Data Platform

    Senior Software Engineer, Data Platform

    Jobright.aiSan Francisco, CA, United States
    Full-time
    Senior Software Engineer, Data Platform.Be among the first 25 applicants.Senior Software Engineer, Data Platform.Parafin is dedicated to supporting small businesses by providing them with essential...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Software Engineer, Data Localization

    Senior Software Engineer, Data Localization

    CloudFlareSan Francisco, CA, United States
    Full-time
    At Cloudflare, we are on a mission to help build a better Internet.Today the company runs one of the world's largest networks that powers millions of websites and other Internet properties for cust...Show moreLast updated: 3 days ago
    • Promoted
    Senior Software Engineer- Database Infrastructure

    Senior Software Engineer- Database Infrastructure

    DiscordSan Francisco, CA, United States
    Full-time
    Discord is used by over 200 million people every month for many different reasons, but there's one thing that nearly everyone does on our platform : . Over 90% of our users play games, spending a comb...Show moreLast updated: 12 days ago
    • Promoted
    Senior Software Engineer, Data

    Senior Software Engineer, Data

    Anvilogic, Inc.Palo Alto, CA, United States
    Full-time
    Anvilogic is a Palo Alto-based AI cybersecurity startup founded in 2019 by security veterans and data scientists from Fortune 500 companies. Our mission is to democratize threat detection and huntin...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Senior Software Engineer, Data

    Senior Software Engineer, Data

    DoximitySan Francisco, CA, United States
    Full-time
    Doximity is transforming the healthcare industry.Join our mission to help every physician be more productive and provide better care for their patients. As medicine's largest network in the United S...Show moreLast updated: 2 hours ago