Apollo / Lightwave Fabrics: How Google Used OCS to Rewrite the Data Center Network
Introduction
In data center networking (DCN), Clos + electronic packet switching (EPS) was once the only answer. Then AI/ML clusters went mainstream, east-west traffic exploded, and every speed upgrade pushed power and cost higher — the old approach of "stack more electronic switch chips, pull more copper and patch cords" started to look clunky and expensive.
Google's Apollo / Lightwave Fabrics made a pivotal choice: put optical circuit switching (OCS) into the production network for real, making "reconfigurable topology" a day-to-day operation rather than a lab demo. This doesn't throw EPS away; instead it uses OCS + SDN control + evolving WDM optics to redraw which part is handled in the optical domain and which in the electrical domain, putting network upgrades and expansion back on an economically and technically sound track.
Below, from an engineering-management and product perspective, I break down the full path: history → architecture and hardware → control and reconfiguration → measured gains → ML applications → risks and next steps.
Contents
1. History (Timeline): From Idea to Production Scale
SIGCOMM 2022 "Jupiter Evolving"
The key turn: Jupiter moved from Clos to a direct-connect topology, with a middle layer of MEMS OCS forming the DCNI (Datacenter Interconnection Layer), making topology reconfiguration and incremental upgrades a routine process rather than a one-off overhaul.
Published results: 5× speed/capacity, −30% CapEx, −41% power, achieved under real production traffic; OCS reconfiguration is 3× faster than the old patch-panel process; in live operation 60% of traffic takes direct paths, averaging just 1.4 block hops.
Significance: it proved that OCS + SDN can replace the traditional spine layer, retaining high capacity while leaving room for upgrades and operations. (Google's research page summary also corroborates these figures and the methodology.)
2022 "Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale"
The first system-level white paper, fully disclosing the design motivations and trade-offs behind the 3D-MEMS Palomar OCS, bidirectional links using circulators, and coexistence of multiple WDM module generations.
Key point: circulators make a single fiber "full-duplex," effectively doubling OCS radix; the paper details how the resulting MPI/RIN/return-loss interference is handled and the specs it demands.
OFC 2023 "Apollo: Large-Scale Deployment of OCS"
Hardware details and volume-deployment experience took center stage:
Palomar OCS: 136×136 non-blocking, insertion loss < 2 dB, typical return loss ≈ −46 dB (spec < −38 dB), system power max 108 W.
WDM optics evolution: the cost and power-efficiency curve from 40G QSFP+ → 800G OSFP; DSP-based PAM4 and OIM (Optical Interference Mitigation) counter the interference introduced by the bidirectional circulator path.
Three core capabilities: Topology Engineering, Fabric Expansion, Rapid Tech Refresh (coexistence of multiple module/ASIC generations).
SIGCOMM 2023 "Lightwave Fabrics"
Abstracts the OCS concept into a Lightwave Fabric serving both datacenter + ML scenarios; for very large clusters such as TPU v4, it reconfigures topology according to communication patterns, using shorter average paths and higher effective throughput to serve long-lived elephant flows.
Google Cloud Blog (Jupiter series)
A product-level summary: in production networks, flow completion time ~−10%, throughput ~+30%, power ~−40%, cost ~−30%; while also enabling live reconfiguration and coexistence of heterogeneous speeds, avoiding the old "upgrade means major surgery" problem. Google Cloud
In 2024 Google again reviewed the Jupiter generations (5th-gen Jupiter at 13 Pb/s bidirectional bisection bandwidth), showing the same path is still expanding.
Plain-language explainer:
Think of a data center as the road network of a huge city. The old "Clos architecture" is a web of intersecting highways, with traffic directed by traffic lights (electronic switches). But as AI traffic explodes, relying on traffic lights alone means more congestion and more power.
Starting in 2022, Google rolled out Project Apollo, bringing in "light paths" as express lanes so the big trucks (huge flows such as AI training) get dedicated lanes instead of squeezing in with small cars (short flows). The journey went from research (Mission Apollo) → deployment (SIGCOMM 2022, OFC 2023) → ML applications (Lightwave Fabrics), steadily turning a lab technology into everyday infrastructure.
2. Architecture Overview: Clos → Direct-Connect (OCS + SDN + WDM)
The core design removes the "switching function of the spine" and instead uses OCS (optical cross-connects) to provide light paths between ABs (Aggregation Blocks), with an SDN control plane (called Orion internally at Google) handling traffic engineering (TE) and topology engineering (ToE).
DCNI layer: an optical switching middle layer made of multiple 3D-MEMS OCS units, enabling programmatic restriping and logical topology reconfiguration.
Bidirectional via circulators: each light path runs full-duplex on a single fiber, maximizing OCS port efficiency, but this demands strict return-loss specs and OIM mechanisms, so the transceivers use DSP-PAM4 + digital compensation.
WDM + multi-generation coexistence: the OCS, fiber and circulators are highly transparent to data rate and wavelength, so 40G/100G/200G/400G/800G modules can be upgraded in place over the same "optical backbone," amortizing the most expensive layer (passive optics and OCS) over a long life.
Put simply: the closer to the core, the more "optical"; the closer to the edge, the more "electrical." The optical domain handles fast reshuffling of "channels" between nodes; the electrical domain handles packet-level "scheduling/buffering/queuing." This split brings both upgrade cadence and engineering complexity back under control.
Plain-language explainer:
The traditional way: every neighborhood (rack) must go through "downtown" to reach another neighborhood, which means detours. Apollo's way: replace the traffic lights in the middle (spine switches) with optical intersections (OCS) that can directly redraw "which two neighborhoods get a direct fiber expressway." Long-lived, high-volume transfers no longer detour, so efficiency rises. And unlike electronic chips, fiber doesn't need massive power and cooling, making it greener.
3. Hardware Layer: Palomar OCS, WDM and Link Engineering
Palomar OCS (3D-MEMS)
Structure: 2D fiber array + 2D lens array + dual MEMS mirror arrays; each traffic beam bounces off two mirrors to form any port-to-port connection; the OCS is broadband and reciprocal, allowing arbitrary N×N mapping.
Performance: 136×136, IL < 2 dB, typical RL ≈ −46 dB, max 108 W, deployed at large volume (tens of thousands of ports).
Control: camera feedback monitors an 850 nm monitoring beam, cutting the BOM and complexity of per-mirror sensing (critical for maintainability at volume).
WDM and Transceivers (OIM / DSP)
Evolution: 40G QSFP+ → 800G OSFP, 20× bandwidth, while power efficiency improves and unit cost falls (optimized for datacom).
OIM: circulator-based bidirectionality introduces MPI/in-line reflections/crosstalk, handled with high ER requirements and RIN control, plus DSP-PAM4 algorithms for cancellation/equalization.
Link budget: extra loss from OCS + circulator → recovered with high-power DMLs, low-loss mux/demux packaging, and FEC and DSP tuning.
Plain-language explainer:
Palomar OCS is like a giant "magic mirror array": each tiny mirror can tilt to reflect a light beam to a designated exit. It acts as a "maze junction manager" for light, redirecting light paths in milliseconds. WDM (wavelength-division multiplexing) is like a multi-lane highway: one fiber carries many colors of light at once, each color a lane. This lets old 40G, 100G and 400G modules share the same fiber — like running more cars without rebuilding the highway.
4. Control and Reconfiguration: Two-Layer Optimization with TE + ToE
Orion (SDN) splits Jupiter into multiple control domains:
Intra-domain (RE): routing within an AB and its external interfaces;
Inter-domain (DCNI): the OCSes are split into four failure domains (25% each) with a multi-level factorized topology mapping, limiting the blast radius of any single failure, and minimizing the delta of each reconfiguration (also minimizing the capacity that must be drained).
Two optimization objectives:
TE (Traffic Engineering): performs WCMP / load splitting on a given topology, balancing between direct (stretch=1) and indirect (stretch=2) paths; latency-sensitive short flows prefer direct paths, while long flows are capacity-driven.
ToE (Topology Engineering): dynamically changes the number and targets of direct links to match the traffic matrix and blocks of differing speed/size; supports live incremental expansion (1/8 → 1/4 → 1/2 → full DCNI).
Result: in practice 60% of traffic goes direct, with an average block-level path of 1.4 hops, giving a better trade-off between throughput and tail latency (FCT); OCS reconfiguration is 3× faster than the old method, which operations teams appreciate.
Plain-language explainer:
If the OCS is a "shape-shifting fiber road," then the SDN controller (Orion) is the traffic control center.
TE (traffic engineering): like real-time navigation (Google Maps), deciding which roads to split traffic across for speed.
ToE (topology engineering): like a city planner, redesigning the road structure when demand changes — for example, adding a direct expressway.
Working together, they turn the whole network into a city that can be "rebuilt on demand."
5. Quantified Gains and Operational Observations
Hard numbers (published by Google): FCT −10%, throughput +30%, power −40%, CapEx −30%; the network keeps evolving through non-disruptive upgrades, replacing "move-the-whole-data-hall" spine refreshes.
Engineering benefits:
The most expensive layer (optical) becomes a durable asset, with WDM/OCS stretching its life cycle;
Upgrade flexibility: ABs of different speeds can coexist, avoiding "a slow spine throttling fast ABs";
Lower operational risk: topology reconfiguration becomes a defined, predictable process;
AI/ML friendly: for known all-to-all/multi-stage exchange communication patterns, configure the topology first, then launch the workload.
Plain-language explainer:
Based on published data, Apollo's impact is remarkable:
Transfer completion time cut by ~10% (small cars no longer get squeezed by big trucks).
Total throughput up ~30% (smoother roads).
Power down 40% (far fewer power-hungry electronic traffic lights).
Cost down 30% (fiber infrastructure lasts longer).
These numbers are not simulations — they come from data centers Google actually runs. For a company handling global Search, YouTube and cloud computing every day, these savings make a huge difference.
6. Landing in ML/TPU Clusters: Lightwave Fabrics
Scenario: training clusters such as the TPU v4 Pod with thousands to tens of thousands of accelerators, whose traffic patterns are long-lived, predictable and phase-based.
Method: treat the fabric as a "programmable optical layer," reshaping the direct-connect matrix before each training phase to shorten average paths and reduce congestion, effectively increasing compute density.
Significance: this "route first, compute second" mindset means the network is no longer a passive carrier but is co-designed with the workload.
Plain-language explainer:
AI training is not like ordinary small data transfers — it is more like "thousands of trucks all exchanging cargo with each other at the same time."
With the old road design, everyone would jam up downtown. With OCS, Google can redraw the road topology before each training phase begins, giving these trucks dedicated direct expressways. That is the core of Lightwave Fabrics: AI training is no longer limited by network bottlenecks. Google's TPU v4 Pod (thousands of AI chips) relies on this network design to reach exaflop-class compute.
7. Risks and Challenges
Switching latency: OCS still switches on a millisecond scale, unsuitable for per-packet scheduling of short flows → in practice, TE sends short flows over nearby/indirect paths, while ToE reconfigures only for long-lived large flows and phase transitions.
Optical interference and return loss: the price of circulator bidirectionality → specs (RL/ER/RIN) set a higher bar, and transceivers compensate with DSP-PAM4 + OIM.
Bursts beyond steady state: requires defining a reconfiguration cadence and trigger thresholds; keep ToE frequency in a range that is operations-friendly and effective for throughput.
Failure-domain design: if an OCS rack fails → uniform fan-out and four equal failure domains ensure "any single-rack failure is evenly spread," preserving ≥75% capacity.
Open interoperability: to grow into an ecosystem, more standardized OCS control interfaces and multi-vendor interoperability are needed (recent public research is already validating SDN controllers for open OCS).
Plain-language explainer:
Of course, the system has its challenges:
Switching speed: OCS needs milliseconds, too slow for short packets, so electronic switches are still required.
Optical interference: single-fiber bidirectional links introduce reflections and noise, which must be corrected by more powerful DSP (digital signal processing).
Fault tolerance: an optical switch failure must not take down the whole data center, so "failure domains" are designed to isolate problems locally.
Standardization: Google can run it in-house today, but for industry-wide adoption, open control protocols are needed so OCSes from different vendors can interoperate.
Looking ahead, if OCS switching latency can be cut further and control software standardized, this system could become the common foundation of next-generation data centers.
Conclusion
The key value of Apollo / Lightwave Fabrics is not flashy optics, but treating topology as a tunable resource, bringing network upgrades, expansion and AI/ML training back to a predictable engineering process.
OCS + SDN + WDM together push down power, cost, latency and upgrade risk all at once. It feels like "far fewer cables pulled, far fewer boxes moved," yet the system metrics look better.
Comments