Talent.com
Research Scientist - Distributed Machine Learning
Research Scientist - Distributed Machine LearningInstitute of Foundation Models • Sunnyvale, CA, United States
Research Scientist - Distributed Machine Learning

Research Scientist - Distributed Machine Learning

Institute of Foundation Models • Sunnyvale, CA, United States
9 days ago
Job type
  • Full-time
Job description

About the Institute of Foundation Models

We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.

As part of our team, you'll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.

Role Overview

Build and scale distributed pre-training frameworks

  • Set up DeepSpeed / FSDP / Megatron-LM across multi-node GPU clusters.
  • Create robust launch scripts, resilient checkpoints, and job monitoring (e.g. NCCL / GLOO / GPU).

Turn mathematical ideas into fast production code

  • Prototype new optimizers or attention methods (like in PyTorch / NumPy / JAX orothers ) .
  • Convert them into efficient CUDA / Triton kernels with custom gradients and tests.

    • Boost training efficiency and stability

  • Lead mixed-precision training, push bf16, fp8, etc, into daily runs, track their accuracy-vs-speed gains, and be able to analyze numeric stability
  • Apply kernel fusion, communication tuning, and memory optimization to reach state-of-the-art throughput.

    • Accelerate research velocity

  • Build logging, metrics, and other experiment-tracking tools for rapid iteration.
  • Design ablation studies and statistical tests that validate-or refute-new ideas.
  • Mentor interns and junior engineers through clear async design docs and code reviews.

    • You'll work side-by-side with researchers, ship production code, and shape the future of large language models.

    Why You'll Love This Job

  • Frontier-scale impact - Train and ship cutting-edge models powering MBZUAI research and industry collaborations.
  • Research x Engineering blend - Move breakthrough papers into real systems and publish your own results.
  • End-to-end mastery - Touch everything from petabyte data loaders to custom low-level kernels-experience that's rare elsewhere.
  • Open, mission-driven science - Join a transparent culture tackling problems that truly advance AI.
  • Founding-team growth - Help set direction for IFM U.S. and lead the next generation of AI development.

    • Key Responsibilities

  • Framework Ownership - Productionize a PyTorch / JAX pre-training stack and keep it reliable at scale.
  • Custom Optimizer Implementation - Code new algorithms in distributed frameworks directly from mathematical specs.
  • Experiment Infrastructure - Build reusable modules, logging, and metrics dashboards that speed up research cycles.
  • Performance Optimization - Apply kernel fusion, communication optimization, and memory management to thousands of GPU jobs.
  • Distributed Debugging - Rapidly diagnose gradient synchronization, collective-ops, or fault-tolerance issues.
  • Collaboration - Document designs clearly, run post-mortems, and partner with global research teams.

    • Qualifications

    Must-Haves

  • 5 + years combined industry or hands-on research experience with large-scale deep-learning training.
  • Led at least one large-scale transformer pre-training run
  • Expert PyTorch or JAX / Flax plus DeepSpeed, FSDP, Megatron-LM, or MosaicML Composer .
  • Experience with distributed training at scale (100+ GPUs).
  • Proven multi-node GPU work (Slurm, K8s, or Ray) and NCCL / GLOO debugging.
  • Strong software engineering skills on large ML codebases
  • Ownership of mixed- or low-precision paths ( bf16, fp8, 4-bit ) with accuracy validation.
  • Clear written communication (design docs, RFCs, post-mortems).

    • Nice-to-Haves

  • NeurIPS / ICML / ICLR papers or open-source contributions to major ML frameworks.
  • Experience implementing optimization algorithms (e.g., SGD variants, Adam, second-order methods).
  • Background in numerical computing.
  • Ability to translate math and
  • build high-perf CUDA / Triton kernels.

    • $150,000 - $450,000 a year

    Total compensation target : $300 K - $600 K (base salary + target bonus up to 30 %), commensurate with experience.

    Visa Sponsorship

    This position is eligible for visa sponsorship.

    Benefits Include

  • Comprehensive medical, dental, and vision benefits
  • Bonus
  • 401K Plan
  • Generous paid time off, sick leave and holidays
  • Paid Parental Leave
  • Employee Assistance Program
  • Life insurance and disability
    • Create a job alert for this search

    Research Scientist Distributed Machine Learning • Sunnyvale, CA, United States

    Similar jobs
    Research Scientist - Machine Learning System

    Research Scientist - Machine Learning System

    ByteDance • San Jose, CA, United States
    Full-time
    AML-MLsys combines system engineering and the art of machine learning to develop and maintain massively distributed ML training and Inference system / services around the world, providing high-perfor...Show more
    Last updated: 9 days ago • Promoted
    Research Scientist, Machine Learning (PhD)

    Research Scientist, Machine Learning (PhD)

    META • Sunnyvale, CA, United States
    Full-time
    Meta is at the forefront of a transformative change in its business and technology, and our research teams are driving this evolution. As a Research Scientist at Meta, you will have the opportunity ...Show more
    Last updated: 8 days ago • Promoted
    Senior Machine Learning Research Scientist

    Senior Machine Learning Research Scientist

    Arc Institute • Palo Alto, CA, United States
    Full-time
    Machine Learning Research Scientist.The Arc Institute is a new scientific institution that conducts curiosity-driven basic science and technology development to understand and treat complex human d...Show more
    Last updated: 30+ days ago • Promoted
    Senior Applied Deep Learning Research Scientist, Efficiency

    Senior Applied Deep Learning Research Scientist, Efficiency

    NVIDIA • Santa Clara, CA, United States
    Full-time
    We are now looking for an Applied Deep Learning Research Scientist, Efficiency!.Join our ADLR – Efficiency team to make deep learning faster and consume less energy! Our team influences the next-ge...Show more
    Last updated: 8 days ago • Promoted
    Research Scientist, Machine Learning for Monetization (PhD)

    Research Scientist, Machine Learning for Monetization (PhD)

    META • Sunnyvale, CA, United States
    Full-time
    From making valuable connections between people and businesses to building premium services that deliver high-value experiences, the Monetization organization at Meta empowers people and businesses...Show more
    Last updated: 8 days ago • Promoted
    Research Scientist 4 Machine Learning and Inference Research LLM PostTraining

    Research Scientist 4 Machine Learning and Inference Research LLM PostTraining

    Netflix • Los Gatos, CA, United States
    Full-time
    Netflix is one of the world's leading entertainment services, with over 300 million paid memberships in over 190 countries enjoying TV series, films and games across a wide variety of genres and la...Show more
    Last updated: 8 days ago • Promoted
    Research Scientist Privacy-Preserving Large-Scale Model Training & Architecture Optimization

    Research Scientist Privacy-Preserving Large-Scale Model Training & Architecture Optimization

    Tik Tok • San Jose, CA, United States
    Full-time
    At TikTok, we treat privacy as our top priority in our product design and implementation.Privacy is not just about regulation compliance, but also about a more trusted way to enable technology inno...Show more
    Last updated: 9 days ago • Promoted
    Research Scientist, Recommendation Systems

    Research Scientist, Recommendation Systems

    DeepMind • Mountain View, CA, United States
    Full-time
    Our team operates at the frontier of modern recommender systems.With a proven track record of innovating and deploying novel deep learning algorithms and systems at scale, we are currently focused ...Show more
    Last updated: 8 days ago • Promoted
    Senior Research Scientist - Machine Learning System

    Senior Research Scientist - Machine Learning System

    ByteDance • San Jose, CA, United States
    Full-time
    The Machine Learning (ML) System sub-team combines system engineering and the art of machine learning to develop and maintain massively distributed ML training and Inference system / services around ...Show more
    Last updated: 9 days ago • Promoted
    Research Scientist, RL for Autonomous Planning & World Modeling

    Research Scientist, RL for Autonomous Planning & World Modeling

    Waymo • Mountain View, CA, United States
    Full-time
    Waymo is an autonomous driving technology company with the mission to be the world's most trusted driver.Since its start as the Google Self-Driving Car Project in 2009, Waymo has focused on buildin...Show more
    Last updated: 8 days ago • Promoted
    Senior ML Research Scientist, Perception Foundation Encoder

    Senior ML Research Scientist, Perception Foundation Encoder

    Nuro • Mountain View, CA, United States
    Full-time
    Nuro is a self-driving technology company on a mission to make autonomy accessible to all.Founded in 2016, Nuro is building the world's most scalable driver, combining cutting-edge AI with automoti...Show more
    Last updated: 30+ days ago • Promoted
    Senior Machine Learning Scientist

    Senior Machine Learning Scientist

    Expedia, Inc. • San Jose, CA, United States
    Full-time
    Senior Machine Learning Scientist.Expedia Group brands power global travel for everyone, everywhere.We design cutting-edge tech to make travel smoother and more memorable, and we create groundbreak...Show more
    Last updated: 30+ days ago • Promoted
    Research Scientist 4 - Machine Learning and Inference Research (LLM Post-Training)

    Research Scientist 4 - Machine Learning and Inference Research (LLM Post-Training)

    Netflix • Los Gatos, CA, United States
    Full-time
    At Netflix, our mission is to entertain the world.Together, we are writing the next episode - pushing the boundaries of storytelling, global fandom and making the unimaginable a reality.We are a dr...Show more
    Last updated: 9 days ago • Promoted
    Research Scientist, Large Recommender Models-TikTok Recommendation

    Research Scientist, Large Recommender Models-TikTok Recommendation

    Tik Tok • San Jose, CA, United States
    Full-time
    Team Introduction You'll be joining the TikTok Recommendation team focusing on advancing large-scale recommender systems that power TikTok's personalized content discovery and user experiences.By d...Show more
    Last updated: 7 days ago • Promoted
    ML Research Scientist

    ML Research Scientist

    Apple • Santa Clara, CA, United States
    Full-time
    Our team is looking for a researcher to work with a collaborative group of ML and robotics specialists within Apple’s organization. This team is working to leverage foundation models and build upon ...Show more
    Last updated: 8 days ago • Promoted
    Senior Research Scientist, Foundation Model (LLM / VLM)

    Senior Research Scientist, Foundation Model (LLM / VLM)

    Waymo • Mountain View, CA, United States
    Full-time
    Waymo is an autonomous driving technology company with the mission to be the world's most trusted driver.Since its start as the Google Self-Driving Car Project in 2009, Waymo has focused on buildin...Show more
    Last updated: 9 days ago • Promoted
    Principal Machine Learning Scientist - Trajectory Generation

    Principal Machine Learning Scientist - Trajectory Generation

    General Motors • Mountain View, CA, United States
    Full-time
    At General Motors, our product teams are redefining mobility.Through a human-centered design process, we create vehicles and experiences that are designed not just to be seen, but to be felt.We’re ...Show more
    Last updated: 30+ days ago • Promoted
    Lead ML Research Scientist

    Lead ML Research Scientist

    Nuro • Mountain View, CA, United States
    Full-time
    Nuro is a self-driving technology company on a mission to make autonomy accessible to all.Founded in 2016, Nuro is building the world's most scalable driver, combining cutting-edge AI with automoti...Show more
    Last updated: 8 days ago • Promoted