Opening. This role is to ensure our frontier models train reliably, efficiently, and at scale. This is demanding, high-impact work that requires both deep technical expertise and a genuine passion for the craft of large-scale ML systems.
What you'll do
This role lives at the boundary between research and engineering. You'll work across our entire production training stack: performance optimization, hardware debugging, experimental design, and launch coordination.
- Own critical aspects of our production pretraining pipeline, including model operations, performance optimization, observability, and reliability
- Debug and resolve complex issues across the full stack—from hardware errors and networking to training dynamics and evaluation infrastructure
- Design and run experiments to improve training efficiency, reduce step time, increase uptime, and enhance model performance
- Respond to on-call incidents during model launches, diagnosing problems quickly and coordinating solutions across teams
- Build and maintain production logging, monitoring dashboards, and evaluation infrastructure
- Add new capabilities to the training codebase, such as long context support or novel architectures
- Collaborate closely with teammates across SF and London, as well as with Tokens, Architectures, and Systems teams
- Contribute to the team's institutional knowledge by documenting systems, debugging approaches, and lessons learned
What you need
- Have hands-on experience training large language models, or deep expertise with JAX, TPU, PyTorch, or large-scale distributed systems
- Genuinely enjoy both research and engineering work—you'd describe your ideal split as roughly 50/50 rather than heavily weighted toward one or the other
- Are excited about being on-call for production systems, working long days during launches, and solving hard problems under pressure
- Thrive when working on whatever is most impactful, even if that changes day-to-day based on what the production model needs
- Excel at debugging complex, ambiguous problems across multiple layers of the stack
- Communicate clearly and collaborate effectively, especially when coordinating across time zones or during high-stress incidents
- Are passionate about the work itself and want to refine your craft as a research engineer
- Care about the societal impacts of AI and responsible scaling
