TorchTitan Update: New Frontier Models, Faster MoE Training and TitanRL for Reinforcement Learning
PyTorch's TorchTitan adds Qwen3.5, Kimi K3 and DeepSeek-V4, an RL stack whose trainer and generator can produce bitwise-identical log-probabilities, and FlexShard, which cut Muon optimizer time by up to 79%.

TorchTitan, the PyTorch team’s native platform for large-scale pre- and post-training, has had a big year. The latest update adds support for a batch of frontier open-weight models, a reinforcement learning stack called TitanRL, and a set of performance features aimed at mixture-of-experts training at scale [1]. For teams choosing a training framework, the headline is that TorchTitan now covers the path from pretraining to RL in one PyTorch-native codebase.
Why it matters
Many RL stacks keep two implementations of the model: one for training and one for inference. Architecture changes have to stay aligned across both, and when a run goes wrong it is hard to tell whether the cause is the reward, the algorithm or the two copies quietly disagreeing [1:1]. TitanRL keeps the same model from pretraining through post-training and inside vLLM [1:2]. Combined with a batch-invariant mode that makes trainer and generator agree exactly, it removes one source of noise from RL experiments.
What happened
New models. TorchTitan added PyTorch-native support for Muse Spark Open, Muse Glimmer, Qwen3-VL, Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4, as full training-stack integrations across its 5D parallelism stack. A new configuration system built on typed Python recipes lets users change parallelism, model components or hardware-specific kernels without forking the model [1:3].
TitanRL. Rollouts run asynchronously, with training and generation each scaling independently. The team reports stable training for hundreds of steps on MoE and dense models, including Qwen3.5-35B-A3B on SWE-Rebench with 64K context on 40 GB300 GPUs [1:4]. In supported on-policy configurations, batch-invariant mode makes the trainer and generator produce bitwise-identical log-probabilities, so the gap between them is exactly zero at every step [1:5].
Ecosystem. TorchTitan is now a training backend in veRL and Miles, the default backend for Meshy, and the inspiration for the training engine in AReaL [1:6].
How it works: the performance features
- local_compile applies
torch.compileto selected regions rather than the whole graph or block, keeping numerics intact without custom kernels [1:7]. - torch_remat is a new activation-checkpointing API with fine-grained control over what is recomputed [1:8].
- Dist-MoE, a newly open-sourced library, moves tokens between experts over symmetric memory and fuses that communication with expert compute on NVIDIA SM100+ GPUs, while staying CUDA-graphable [1:9].
- FlexShard is a new data-parallel sharding library with placements FSDP2 cannot express. Its Owned and BlockShard placements let the distributed Muon optimizer skip all-gathers, cutting optimizer time by up to 79% on DeepSeek-V3 at 256 H100s. BlockShard also aligns shards with FP8 quantization blocks, so quantized parameters can be gathered without extra collectives, improving step time by up to 20% on Llama 3 405B at 256 H100s [1:10].
- GraphTrainer offers a graph-mode training path whose numerical results match eager training bitwise [1:11].
- torch_checkpointing removes per-save synchronization. On DeepSeek-V3 671B at 512 H100s, average trainer blocking per checkpoint fell from 841.2 ms to 34.1 ms [1:12].
At scale, TorchTitan matched Megatron’s loss convergence on the DeepSeek-V3 MLPerf training benchmark, run in MXFP8 on an internal GB300 cluster [1:13]. PyTorch 2.14 also ships a new opt-in NCCL backend, ported from torchcomms and validated on TorchTitan, with in-place process-group reconfiguration for fault tolerance [1:14].
What data and AI engineers should do
- Use batch-invariant mode when changing RL algorithms or kernels. If rewards differ from your baseline, you can rule out trainer/generator mismatch.
- Try FlexShard if you use Muon or blockwise FP8. Those are the cases where FSDP2’s sharding boundaries cost extra communication.
- Adopt torch_checkpointing for frequent checkpoints at scale, where per-save synchronization used to stall training.
- Start from the repository. Clone
pytorch/torchtitanand follow its instructions; the recipes are plain Python and easy to review [1:15].
Limitations and open questions
Most numbers come from Meta’s own clusters, often on GB300 or H100 at hundreds of GPUs, and several features target NVIDIA Blackwell-class hardware, so results on other setups may differ. Bitwise-identical log-probabilities hold only in supported on-policy configurations [1:16]. The post shows RL curves but does not compare TitanRL’s throughput or cost with other RL frameworks.
Key takeaways
- TorchTitan now supports Qwen3.5, Kimi K3, DeepSeek-V4 and more across its 5D parallelism stack. [1:17]
- TitanRL keeps one model from pretraining to vLLM, and batch-invariant mode gives bitwise-identical log-probabilities in supported setups. [1:18]
- FlexShard cut Muon optimizer time by up to 79% on DeepSeek-V3 and step time by up to 20% on Llama 3 405B. [1:19]
- torch_checkpointing cut average trainer blocking per checkpoint from 841.2 ms to 34.1 ms at 512 GPUs. [1:20]
Discussion
Join the discussion: create a free account or sign in to comment.
No comments yet.