HOT CHIPS 2025: Google Ironwood TPU Rack — When AI Systems Are Designed for Long-Term Scalability

Why Google Started with the "System" Rather Than the TPU Chip

In this case study, Google didn't focus on the Ironwood TPU itself; it started from the evolution of the entire TPU system and super pod, reflecting Google's consistent design philosophy.

• TPU has been a "system-level product" from day one

• The chip is just one component of the system

• The architecture must serve both current and future workloads

• Design cycles assume "usable for many years"

For Google, a chip that doesn't fit naturally into the system is meaningless.

TPU System Evolution Is Fundamentally a Response to Workloads

Google clearly explained the forces driving each TPU generation.

• Model parameter counts keep rising

• Mixture of Experts has become the norm

• Both training and inference need larger scale-up domains

• Memory bandwidth and capacity have become bottlenecks at the same time

Three fundamental pillars are under pressure simultaneously:

• Compute

• Memory

• Network

Ironwood is not a point upgrade; it scales all three together.

From TPUv4 to Ironwood: A Change in System Scale

Google used data to show the leap in system scale.

• TPUv4:

  • About 500 TPUs

  • HBM 32 GB

  • 1.2 TB/s bandwidth

• TPUv5P:

  • About 9,000 TPUs

  • HBM about 96 GB

  • 2.8 TB/s bandwidth

• Ironwood:

  • 9,216 TPUs

  • HBM 192 GB

  • 7.4 TB/s bandwidth

  • Per-chip performance up about 10x

This is not just a performance gain but a qualitative change in "system density."

Ironwood's Basic Building Block Is an Entire Rack

Ironwood's smallest physical and logical unit is not a server but a rack.

• One rack holds 16 TPU PCBA trays

• Each tray carries 4 Ironwood TPUs

• One rack = a 64-TPU 3D Torus building block

The rack is at once:

• A logical scale-up node

• A physical deployment unit

• The basic unit for operations and management

Why Google Sticks with the 3D Torus Architecture

Unlike NVLink's single-tier switch architecture, Google has long used a 3D Torus.

• Each rack is a 4×4×4 3D Torus

• ICI (Inter-Chip Interconnect) is the core scale-up network

• The architecture natively supports:

  • Model parallelism

  • Pipeline parallelism

  • Expert sharding

This isn't about simplicity; it's about topology flexibility.

Hybrid Interconnect: PCB, Copper and Fiber Side by Side

Ironwood takes a hybrid approach to interconnect.

• PCB traces: the shortest in-rack distances

• Passive copper cable: in-rack and adjacent links

• Optical fiber: cross-rack scale-up and OCS

This lets the system switch topologies freely across different scales.

• Not locked into a single interconnect technology

• Reconfigurable to match workloads

The Optical Circuit Switch Is the Key Extension of Scale-up

Ironwood doesn't scale on a fixed topology alone.

• OCS racks act as scale-up extension nodes

• They let multiple racks combine dynamically into a super pod

• Scale-up across 144 racks is supported

This lets Ironwood support both:

• Large single training jobs

• Many fragmented jobs running in parallel

An Ironwood Super Pod Is Essentially a Supercomputer

A full Ironwood super pod includes:

• 144 TPU racks

• Multiple OCS racks

• Dedicated CDU (Coolant Distribution Unit) racks

• Separate network racks

Overall scale is roughly:

• 9,216 TPUs

• About 43 exaFLOPS of compute

• A memory pool of nearly 1.8 PB of HBM

This is a supercomputer-class system.

Why Google's Racks Are Noticeably Wider

Ironwood's non-standard rack width is no accident.

• Better front-side serviceability

• More room for cable routing

• Support for high-density liquid cooling manifolds

• Lower cabling and maintenance risk

Generations ago, Google was already trading density for "operability."

Liquid Cooling Is Not an Option but a Prerequisite

Google bet on liquid cooling early.

• Dennard scaling has ended

• Rising power every generation is inevitable

• High-density copper interconnect requires short distances

Liquid cooling solves three problems at once:

• Heat transfer efficiency

• System density

• Control of interconnect distance

Power Management Has Become a System-Level Problem

Ironwood places special emphasis on power behavior.

• Per-rack power exceeds 100 kW

• Dual power domains

• Support for rack-level power capping

Even more important is "dynamic power."

• Synchronous training can cause 10–15 MW power swings

• Hardware and software must coordinate control

• Fast regulation on the order of 100 ms

This is no longer simply a power supply design problem.

Turn-up Velocity Is a Hidden but Critical Design Metric

Google repeatedly mentioned one key term: turn-up velocity.

• Can the system come online quickly?

• Can it be deployed smoothly in the data center?

• Can it be debugged and repaired quickly?

To that end, Ironwood's rack level is designed with:

• Pre-installed fiber and copper cabling

• Clear cable management

• Highly modular tray design

Google Cares Not About Point Performance but About Long-Term, Stable Scaling

Ironwood's design philosophy is very clear.

• The architecture must support future workloads

• The system must remain usable for many years

• Hardware and software evolve together, not in one direction

This is also the core reason Google keeps building its own TPU systems.

Simple Tech Trend View

Ironwood demonstrates a different way of thinking about AI systems.

• NVIDIA is solving "the engineering problem of extreme density"

• Meta is solving "deployability in existing data centers"

• Google is solving "a system architecture that scales for the long term"

In Google's world,

an AI system is not a product but long-term infrastructure.

Ironwood was not built for one generation of models; it leaves room for "models that don't exist yet."