Site Reliability Engineer - Storage

Job Description

Job Description

About xAI

xAI's mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company's mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All engineers are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

About the Role

As a Site Reliability Engineer (SRE) : Storage at xAI, you will play a pivotal role in ensuring the reliability, scalability, and performance of our petabyte-to-exabyte scale storage infrastructure, including filesystems and our internal storage product supporting the Colossus superclusters in Memphis — the world's largest AI training clusters with hundreds of thousands of liquid-cooled GPUs. We're deploying multiple exabytes of storage this year across several sites to fuel Grok's training and advanced AI workloads. You will collaborate with storage engineers, software engineers and hardware storage teams to deploy, troubleshoot, and optimize storage for 24 / 7 AI I / O demands like checkpointing and dataset streaming, long term archival storage, and ensure maximum uptime. This is a hands-on technical position in a dynamic environment, offering the opportunity to tackle complex challenges at the intersection of storage systems, hardware integration, and reliability engineering.

Responsibilities

• Deploy, maintain, and scale exabyte-scale storage clusters with a focus on observability, zero-downtime upgrades, and integration with high-density GPU environments.
• Troubleshoot production storage issues across hardware-software stacks : NVMe / PCIe / RDMA paths, firmware bugs, BMC logs, disk failures—performing root cause analysis and automating preventions.
• Collaborate with storage teams to validate server specs, debug field problems and influence custom designs with vendors for cutting-edge AI storage.
• Evaluate and onboard new storage vendors and technologies; benchmark for cost, density and GPU-direct performance against AI training I / O patterns.
• Support storage SDEs by translating engineering requirements into reliable, observable systems; develop scripting and playbooks to reduce toil and enable self-service.
• Lead hardware refreshes for legacy X storage fleets, including migration, decommissioning, and designing repeatable processes for customized solutions.
• Participate in on-call rotations (follow-the-sun, generous stipend) for storage domains; respond to incidents, drive post-mortems, and forecast capacity for EiB+ growth.
• Create and maintain documentation, standard operating procedures, and monitoring for storage health in massive-scale AI pipelines.

Required Qualifications

xAI is an equal opportunity employer.

California Consumer Privacy Act (CCPA) Notice