OCP Global Summit 2025 | Meta: Networking for AI Scaling Up, Out, and for the Future

Oct 5

4 min read

Introduction

At OCP 2025, Meta shared its deep thinking on network architecture for the AI era — from its global network footprint and data center design to the interconnect topology of AI clusters — stressing that "Networking is everywhere in AI." As GPU clusters leap from thousands to millions of GPUs, the network is no longer just support infrastructure; it is the hub of the AI compute architecture. Meta's vision is to build an "open, modular, cross-generation-compatible AI network foundation" so that every layer — from reticle to rack to region — can flexibly combine scale-up and scale-out.

Content

1. The Network Is the Backbone of AI: From the Globe to the Chip

Using product traffic from its 3.5 billion users as an example, Meta showed that behind every interaction are countless "arrows" representing network links. From server-to-server connections inside the data center to intercontinental fiber backbones, every layer depends on a fast, low-latency, scalable network foundation.

Global network footprint:
Meta has built a worldwide POP and cache network, linking continents via subsea cables (such as 2Africa and the Water series) so AI applications can run locally.
Fiber matters as much as power:
Meta noted that "building fiber takes almost as long as building a data center," so beyond power, connectivity is also a key bottleneck for AI infrastructure.

2. From DSF to NSF: The Evolution of Meta's Scale-Out Network

Last year Meta introduced the Disaggregated Scheduled Fabric (DSF), a scheduled architecture managing AI clusters of about 4,000 GPUs. This year, Meta announced two major updates:

Architecture	GPU capacity	Characteristics	Notes
DSF (Distributed Scheduled Fabric)	4,000~20,000	Distributed network with scheduling control	First introduced last year
NSF (Non-Scheduled Fabric)	>20,000	Based on shallow-buffer Ethernet switches	Next-generation core architecture

NSF uses Broadcom 51.2T Ethernet switch silicon with a shallow-buffer design, dropping centralized scheduling in favor of lower latency and flexible load balancing to support clusters of tens of thousands of GPUs. It is already in production at Meta's Ohio data center in the US and is integrated with NVIDIA Spectrum 4 and the FBOSS software stack.

3. Moving Toward Gigawatt-Class Clusters

Meta's data centers are scaling toward gigawatt-class AI clusters. That means the AI systems in a single campus will draw power on the scale of a city. To support that much compute, Meta treats the entire data center network as one "Super Rack."

Key features include:

Liquid-cooled network racks
Hybrid cabling of high-density fiber and short-reach copper
Inter-building AI cluster interconnect (Inter-building AI Training Fabric)

These systems support not only model training but also post-processing and cross-region data synchronization, extending AI compute from a single data center to an entire region.

4. Open Standardization of Scale-Up Networking: The ESUN Initiative

At this year's OCP Summit, Meta formally announced it is leading the launch of the Ethernet for Scale-Up Networking (ESUN) initiative. It is an open project for high-bandwidth in-rack interconnect, aiming to define features at the Ethernet MAC/PHY layers that support low-latency, high-reliability, AI-specific traffic.

The layered architecture is as follows:

Layer	Project	Function	Key participants
Transport Layer	SUE (Scale-Up Ethernet)	Defines the transport protocol between GPUs/XPUs	Broadcom, Meta
Ethernet Layer	EON (Ethernet Open Network)	Optimizes Ethernet framing and flow control	Meta, Cisco, Arista
System Integration	ESUN (Ethernet for Scale-Up Networking)	Provides a unified open industry standard	Led by Meta, in collaboration with OCP

The initiative aims to spare every GPU/ASIC vendor from developing its own proprietary interconnect fabric. Meta wants to make Ethernet the universal language of scale-up, replacing closed interconnects such as NVLink and Infinity Fabric.

5. Open Hardware and Software Ecosystem

Meta's open approach shows up not only in architecture but also at the system level:

Hardware:
- Minipack3N (using the NVIDIA Spectrum 4 ASIC)
- Minipack3 series switches
- Liquid-cooled high-density rack design
Software:
- FBOSS (open-source switch operating system)
- OpenBMC (server management)
- OCP SAI (Switch Abstraction Interface)

All of these are integrated into Meta's Non-Scheduled Fabric (NSF),

forming the network foundation of next-generation AI clusters.

Summary

Meta's network blueprint highlights the core turning point for AI infrastructure over the next decade:

from "a single data center" to a "cross-region AI fabric."

"Networking is the hidden enabler of AI. Every arrow is a network."

Through open architectures (ESUN, SUE, EON), Meta is making Ethernet not only the backbone of scale-out but also the universal interconnect language of scale-up, redefining the boundaries and standards of AI networking.

Extended Perspectives

1. Technology evolution: Ethernet from fabric to silicon

Meta's strategy marks an era of vertical integration for Ethernet:

In the past, Ethernet was purely a networking-layer technology;
now it is reaching down into the chip interconnect layer (scale-up) and up into regional AI fabrics (scale-across). This means Ethernet will become a unified interface "from chip to globe."

2. Supply chain view: Meta's multi-generation coexistence strategy

Meta specifically highlighted the challenge of fleet management: different generations of GPUs, NICs, switches and optics must coexist. This pushes Meta toward a multi-layer open platform model that lets Broadcom, NVIDIA, Cisco and Arista all plug into its ecosystem.

3. Market implication: open scale-up is the next battleground

OCP 2025 made it clear:

Scale-up Ethernet will be the industry's next focus after CPO. Vendors are competing to define the standard interconnect inside the AI rack. By moving first with an open standard, Meta has gained a key voice in the coming rack-scale AI fabric battle.