Talent.com
Senior AI Engineer - APM Experiences
Senior AI Engineer - APM ExperiencesDatadog • Syracuse, New York, United States
Senior AI Engineer - APM Experiences

Senior AI Engineer - APM Experiences

Datadog • Syracuse, New York, United States
7 days ago
Job type
  • Full-time
Job description

The opportunity

Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster. In this role, you will lead end‑to‑end development of LLM- and Agent‑based features that can :

  • Debug and investigate application performance issues down to the root cause, as both a developer assistant and a fully autonomous agent
  • Proactively recommend performance and reliability-based optimizations to prevent the next incident
  • Automatically create intelligent monitors and SLOs for the most important business flows and critical paths

This is a highly product‑minded engineering role : you’ll work from problem discovery and UX all the way to reliable, scalable production systems.

What you’ll do

  • Shape AI experiences for APM. Design and ship LLM / agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes.
  • Own the full loop. Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability.
  • Build robust agent systems. Develop tools, retrieval and planning strategies, and guardrails; manage prompts / evals; design fallbacks and human‑in‑the‑loop paths.
  • Integrate with Datadog’s platform. Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI.
  • Partner deeply. Collaborate with PM, Design, and partner teams to build cohesive experiences.
  • Raise the bar on engineering. Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems.
  • Who you are

    Product‑minded engineer who ships AI to production

  • 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance
  • Proven experience delivering LLM / agent features to production (prompting, tooling, evals, safety / guardrails)
  • Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics
  • Strong ML / applied science fundamentals

  • Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals)
  • Experience choosing / modeling the right technique for the job (e.g., anomaly detection, ranking / recommendation, NLP), and knowing when a heuristic beats a model
  • Fluency with offline / online evals for AI systems; can build reliable golden sets and automatic regressions
  • Distributed systems & observability savvy

  • Experience with microservices performance : tracing, latency breakdowns, concurrency, and resiliency patterns
  • Proficient in Go, Java, or Python; strong API / service design; production ops (monitoring, alerting, on‑call rotation)
  • Nice to have

  • Hands‑on with distributed tracing stacks (OpenTelemetry / Datadog APM), profilers, and logs / metrics pipelines
  • Exposure to planning / agent frameworks, tool‑use orchestration, RAG, and retrieval / indexing for observability data
  • Familiarity with SLO / SLA practices and incident response
  • Benefits and Growth :

  • Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development.
  • Have a lot of influence on product direction and impact on the business.
  • Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn.
  • Competitive global benefits.
  • Continuous professional development.
  • Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

    Create a job alert for this search

    Senior Engineer Ai • Syracuse, New York, United States

    Related jobs
    Associate Dentist

    Associate Dentist

    Aspen Dental • Cicero, NY, US
    Full-time +1
    This opening is for an Associate Dentist.At Aspen Dental, we put You 1st, offering the financial security and job stability that comes with working with a world-class Dental Service Organization (D...Show more
    Last updated: 30+ days ago • Promoted
    Senior Staff RF Electrical Engineer

    Senior Staff RF Electrical Engineer

    Saab • Central Square, NY, US
    Full-time
    Job Description : We are seeking a Level 4 Senior Staff RF Engineer with a strong systems engineering focus to lead the architecture, analysis, and optimization of advanced RF systems for radar and ...Show more
    Last updated: 5 hours ago • Promoted • New!
    Remote Corporate Development Associate - AI Trainer ($50-$60 / hour)

    Remote Corporate Development Associate - AI Trainer ($50-$60 / hour)

    Data Annotation • Syracuse, New York
    Remote
    Full-time +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...Show more
    Last updated: 19 days ago • Promoted
    Travel EP Tech / EP Lab Tech - $1,907 to $2,090 per week in Syracuse, NY

    Travel EP Tech / EP Lab Tech - $1,907 to $2,090 per week in Syracuse, NY

    AlliedTravelCareers • SYRACUSE, NY, US
    Full-time
    AlliedTravelCareers is working with AMN Healthcare Allied to find a qualified Electrophysiology Tech in SYRACUSE, New York, 13210!. Job Description & Requirements.EEG Technologist - (EEG Tech) ....Show more
    Last updated: 30+ days ago • Promoted
    Staff AI Engineer - Notebooks

    Staff AI Engineer - Notebooks

    Datadog • Syracuse, New York, United States
    Full-time
    Datadog Notebooks provide customers with a collaborative surface for ad-hoc data analysis, technical documentation, incident postmortems and runbooks. The power and flexibility of Notebooks also mak...Show more
    Last updated: 30+ days ago • Promoted
    Dashers - Sign Up and Start Earning

    Dashers - Sign Up and Start Earning

    DoorDash • De Ruyter, NY, United States
    Full-time +1
    DoorDash is the #1 category leader in food delivery, food pickup, and convenience store delivery in the US, trusted by millions of customers every day. As a Dasher, you’ll stay busy with a var...Show more
    Last updated: 30+ days ago • Promoted
    CRNA - Anesthesiology job available in Cicero, New York

    CRNA - Anesthesiology job available in Cicero, New York

    Odyssey Staffing • Cicero, NY, US
    Full-time
    Locum CRNAs needed to start ASAP and work ongoing near Cicero, NY.K COMPLETION BONUS - 520 hours worked within 17 consecutive weeks • •. Shift Type : 10-hour shift : 7 : 00 AM – 5 : 30 PM.All ne...Show more
    Last updated: 30+ days ago • Promoted
    Anesthesiologist - Specialists One Day Surgery Center - Full Time

    Anesthesiologist - Specialists One Day Surgery Center - Full Time

    NAPA Management Services Corporation • Syracuse, NY, US
    Full-time
    Anesthesiologist - Specialists One Day Surgery Center - Full Time.Syracuse, New York, United States of America.Syracuse, NY at Specialists One Day Surgery Center!. Specialists' One-Day Surgery C...Show more
    Last updated: 30+ days ago • Promoted
    Air Interdiction Agent

    Air Interdiction Agent

    U.S. Customs and Border Protection • McGraw, NY, US
    Full-time
    Pilot CBP Air Interdiction Agent.Air and Marine Operations (AMO), a component of U.Customs and Border Protection (CBP), offers skilled Pilots interested in law enforcement an opportunity to work wi...Show more
    Last updated: 23 days ago • Promoted
    Restaurant Delivery - Start Earning Quickly

    Restaurant Delivery - Start Earning Quickly

    DoorDash • De Ruyter, NY, United States
    Full-time +1
    DoorDash is the #1 category leader in food delivery, food pickup, and convenience store delivery in the US, trusted by millions of customers every day. As a Dasher, you’ll stay busy with a var...Show more
    Last updated: 30+ days ago • Promoted
    Travel EEG Tech - $2,903 per week in Syracuse, NY

    Travel EEG Tech - $2,903 per week in Syracuse, NY

    AlliedTravelCareers • Syracuse, NY, US
    Full-time
    AlliedTravelCareers is working with Triage Staffing LLC to find a qualified EEG Tech in Syracuse, New York, 13201!.Travel Cardiopulmonary : Neurodiagnostics Syracuse. Shift Details : 8H Days (12 : 00 ...Show more
    Last updated: 20 days ago • Promoted
    Infrastructure Engineer, Unstructured AI

    Infrastructure Engineer, Unstructured AI

    Collibra • Syracuse, New York, United States
    Full-time
    Joining Collibra’s Engineering team .We’re shaping the way some of the largest organizations in the world manage data by helping customers connect the right data and insights for all Data Citizens....Show more
    Last updated: 20 days ago • Promoted
    MRI Technologist

    MRI Technologist

    Crouse Hospital • Syracuse, New York, United States
    Full-time
    Since 1887, Crouse Health has been a leading healthcare provider in Central New York.We’ve combined a family-friendly culture with a passion to provide the best care, which creates an environment w...Show more
    Last updated: 30+ days ago • Promoted
    Travel CT Tech - $3,035 to $3,326 per week in Syracuse, NY

    Travel CT Tech - $3,035 to $3,326 per week in Syracuse, NY

    AlliedTravelNetwork • SYRACUSE, NY, US
    Full-time
    AlliedTravelNetwork is working with AMN Healthcare Allied to find a qualified CT Tech in SYRACUSE, New York, 13215!.Job Description & Requirements. Computed Tomography Technologist - (CT Tech) ....Show more
    Last updated: 4 days ago • Promoted
    Remote Senior FP&A Analyst - AI Trainer ($50-$60 / hour)

    Remote Senior FP&A Analyst - AI Trainer ($50-$60 / hour)

    Data Annotation • Syracuse, New York
    Remote
    Full-time +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...Show more
    Last updated: 19 days ago • Promoted
    Lead Engineer, Applications - Edifecs / QConnect - Remote

    Lead Engineer, Applications - Edifecs / QConnect - Remote

    Molina Healthcare • Syracuse, NY, US
    Remote
    Full-time
    Designs and builds company specific enterprise application systems and technology expertise across multiple disciplines.Applies and promotes key principles (e. Considers business problems “end-to-en...Show more
    Last updated: 10 hours ago • Promoted • New!
    Lead AI Engineer - New York

    Lead AI Engineer - New York

    Distyl Ai • Syracuse, New York, United States
    Full-time
    AI native technologies for humans & AI to collaborate to power the operations of the Global Fortune 1000.In just 24 months, we’ve rapidly grown to partner with some of the world’s largest enterpris...Show more
    Last updated: 30+ days ago • Promoted
    Remote M&A Associate - AI Trainer ($50-$60 / hour)

    Remote M&A Associate - AI Trainer ($50-$60 / hour)

    Data Annotation • Syracuse, New York
    Remote
    Full-time +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...Show more
    Last updated: 19 days ago • Promoted