TorchTPU: PyTorch Runs Natively on Google TPUs, and Hugging Face Models Come Along
Google and Meta built TorchTPU, a native PyTorch backend for TPUs with eager execution and torch.compile. Hugging Face Transformers, Diffusers and Kernels now treat "tpu" as just another device.

Running PyTorch on Google’s TPUs used to mean porting code to PyTorch/XLA or learning JAX. TorchTPU changes that. It is a native PyTorch backend for TPUs, built by Google with Meta’s PyTorch team, and Hugging Face has added support across Transformers, Diffusers and Kernels so a model from the Hub can simply be sent to "tpu" [1] [2]. For teams with PyTorch code and access to TPU capacity, the porting cost drops to roughly a device string.
Why it matters
The old path had real costs. The PyTorch team says PyTorch/XLA, introduced in 2020 and built on lazy tensors, required intrusive code changes to mark compilable regions, suffered severe compile-time latency, lacked eager execution and lagged on new distributed primitives [2:1]. Hugging Face describes the everyday version: tensors were lazy, so code did not always run when you expected, and small changes could trigger recompilations [1:1].
TorchTPU’s goal is that porting a workload is as simple as changing .to("cuda") to .to("tpu"). The team is candid that this gets a model running, not to peak performance, since shapes and sharding may be tuned for GPUs [2:2]. That matters for capacity planning: TPUs become a realistic second source of accelerators for PyTorch shops without a rewrite.
What happened
TorchTPU started at Google in mid-2025 and is open source. The PyTorch team says it is already used in production for inference and training by some of the largest Google Cloud customers [2:3]. At launch it has [2:4]:
- eager and compiled modes, including a new DeferAndFuse eager mode;
- about 450 ATen ops, including forward and backward passes for all core ops needed by major architectures;
- custom kernels via JAX/Pallas and the Helion DSL;
- out-of-the-box execution on the 5,000 most downloaded Hugging Face models;
- validated integration with vLLM, SGLang, TorchTitan, Ray and TorchRec.
Hugging Face says Optimum TPU, its earlier bridge, is no longer needed to run models with Transformers and Diffusers [1:2].
How it works
Eager first. TorchTPU registers as a new PyTorch backend and offers three eager modes. Strict, the default, dispatches one op at a time, asynchronously, like eager PyTorch on a GPU. Debug runs synchronously, like CUDA_LAUNCH_BLOCKING. DeferAndFuse collects ops until a result is needed, then compiles and runs them together; the team reports it can be 50% or more faster than strict eager while lowering peak HBM use [2:5].
Compiled when you want speed. torch.compile is built on TorchDynamo rather than lazy tensors, with compiled graphs cached so identical shapes are not recompiled [2:6].
Distributed the PyTorch way. TPU collectives sit at the Process Group level, so DDP, FSDP and DTensor idioms carry over. TorchTPU uses one process per device, unlike JAX’s one process per host [2:7].
Precision. By default, matrix multiplications use BF16 inputs with FP32 accumulation. Higher-precision modes are available at roughly 3x or 6x the matrix-unit cost [2:8].
In Hugging Face libraries
- Transformers v5: pass
device_map="tpu"to load a model directly onto the TPU. It runs eagerly by default, withtorch.compileand distributed setups available [1:3]. - Diffusers:
pipe.to("tpu")for eager prototyping, plus one compile call for deployment. Models too big for one chip can be sharded at load time with aTensorParallelConfig[1:4]. On a Cloud TPU v6e-4, a 1024x1024 image at 40 steps took 3.5 seconds compiled against 8.0 seconds eager [1:5]. - Kernels: Pallas kernels from the Kernels Hub can be dropped in. With a ragged-dot kernel for mixture-of-experts layers, decoding at batch size 64 was 21.5x faster on Qwen3-30B-A3B and 12.8x faster on Gemma 4 26B-A4B than the default implementation, on a TPU v6e-8 [1:6].
What data and AI engineers should do
- Install with the CPU index:
pip install torch-tpu --extra-index-url https://download.pytorch.org/whl/cpu. Without it, pip pulls the much larger CUDA build of PyTorch [1:7]. - Start eager, then compile. Get the model running, try DeferAndFuse, and add
torch.compilefor deployment. - Keep shapes static when compiling. The Diffusers guide uses
dynamic=False; each new image size compiles once, and the first call is slow [3]. - Compile around CPU offload. For pipelines that need
enable_model_cpu_offload(), compile only the transformer’s repeated blocks, because the offload hooks cannot be traced [3:1]. - Report failures. Hugging Face turns each failure into a repro script and an issue on the TorchTPU repository [1:8].
Limitations and open questions
Support is not complete. Of the Transformers tests that run on TPU, 96.5% pass; Hugging Face has reported 34 issues and 19 are fixed [1:9]. The multi-chip launch command in the Diffusers guide relies on an internal TorchTPU module that is expected to change [3:2]. XLA still assumes shapes are known at compile time; the team is adding bounded dynamism, where only maximum sizes are fixed, but it is described as work in progress [2:9]. And because a TPU can be used by only one process, colocating RL workloads is not supported out of the box yet [2:10].
Key takeaways
- TorchTPU is a native PyTorch backend for TPUs from Google and Meta, already used in production by large Google Cloud customers. [2:11]
- Hugging Face Transformers, Diffusers and Kernels support it: load with
device_map="tpu"or.to("tpu"). [1:10] - The new DeferAndFuse eager mode can be 50% or more faster than strict eager and uses less HBM. [2:12]
- 96.5% of Transformers tests that run on TPU pass so far; expect some rough edges. [1:11]
Sources
Hugging Face Blog, “Hugging Face ❤️ TorchTPU: TPUs, Natively”, 2026-10-20. https://huggingface.co/blog/hf-torchtpu ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
PyTorch Blog, “Introducing TorchTPU”, 2026-10-20. https://pytorch.org/blog/introducing-torchtpu/ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
Hugging Face Diffusers documentation, “TorchTPU”. https://huggingface.co/docs/diffusers/main/optimization/tpu ↩︎ ↩︎ ↩︎
Discussion
Join the discussion: create a free account or sign in to comment.
No comments yet.