OCP Global Summit 2026 | Meta, AMD | Arvind Srinivasan, Sandeep Nagaraja, Balakrishnan Raman | MetaRoCE: Rethinking RDMA for AI

1. Why Meta Rewrote the RDMA Transport Layer

This Meta Expo Hall session ran just 17 minutes, but it addressed the deepest layer of the AI network: RoCEv2 cannot keep up with today's cluster sizes.

The slides set the scene bluntly: AI clusters already span hundreds of thousands of accelerators across multiple data centers and regions, and are being designed for a million; from 2024 to 2030 the number of network-connected accelerators grows roughly 10x. Collectives such as All-Reduce and All-to-All are gated by the slowest node. Meta's Arvind Srinivasan put it this way: collectives are like an army marching, and the whole formation moves only as fast as its slowest soldier.

RoCEv2's problems are well known: it relies on PFC for a lossless fabric, falls back to go-back-N on loss, and pins each connection to one ECMP-hashed path. At scale, PFC deadlock risk, wasted retransmission bandwidth and hash-collision hotspots all show up at once.

Ethernet has been moving from scale-out into scale-up over the past two years (see OCP APAC Summit 2026 | Ethernet Starts Eating Scale-Up), yet the scale-out transport itself was never properly rewritten. MetaRoCE is Meta's answer, with three goals on the slide: more than 90% network efficiency, transparent failover across multiple planes, and operation on commodity Ethernet with full verbs compatibility.

2. Design Trade-offs: The Switch Only Needs ECN and ECMP

What stands out about MetaRoCE is not what it adds, but what it refuses. The slides list two sets of architectural decisions:

• Commodity Ethernet switches only: ECN and ECMP, with no trimming, adaptive routing or in-network telemetry. Meta's reasoning is that trimming adds complexity to both switch and NIC, while selective acknowledgment (SACK) detects loss without either.

• PFC, go-back-N and reorder buffers do not scale: PFC headroom grows with distance and large fabrics risk deadlock; go-back-N wastes bandwidth; NIC reorder buffers consume SRAM that grows with the bandwidth-delay product (BDP).

Meta's alternative fits in one line: order completions, not delivery. Out-of-order packets are written directly to the right place in destination memory, the application only sees messages complete in order, and the NIC needs no reorder buffer at all.

▲ Design trade-offs between RoCEv2 and MetaRoCE: lossless networking, go-back-N, single-path and in-order delivery are all replaced by loss tolerance, selective retransmission, multipath spraying and out-of-order placement; the switch is left with only ECN and ECMP (Image source: Simple Tech Trend original, data from Meta, AMD 2026 OCP Global Summit presentation)

This is a clear departure from the UEC (Ultra Ethernet Consortium) approach. The UEC 1.0 specification also supports packet spraying and out-of-order delivery, but lists packet trimming as an optional switch feature; NVIDIA Spectrum-X relies on switch-side adaptive routing. MetaRoCE asks the least of the switch among the three, and the price is that the NIC makes every decision.

3. Multipath: One QP, Many Paths, Each With Its Own Window

The basic unit in MetaRoCE is the path. When an application posts a message, the protocol chunks it into MTUs and sprays them across all available paths; each path keeps its own path window (PWND) and RTT measurement, and the QP's total window is the sum of all path windows (QWND = ΣPWND).

The design has three effects:

• Congestion only affects one path: an ECN mark or N dropped packets triggers a multiplicative decrease on that path alone, leaving other paths untouched; an ACK on any path can trigger additive increase.

• More paths do not mean more aggression: adding paths adds bandwidth, but because each path is controlled independently, the overall aggressiveness toward the network does not rise.

• One QP carries multiple streams: applications today often open several QPs to aggregate bandwidth, and those QPs interfere with each other, making congestion control hard. MetaRoCE multiplexes streams onto a single QP and leaves multipathing to the layer below.

▲ MetaRoCE uses the path as its basic unit: each path has its own window and RTT, the QP's total window is the sum across paths, and a new path takes over when one fails (Image source: Meta | 2026 OCP Global Summit presentation (p. 6))

The other key piece is receiver rate hints (RCN). In an incast, where many senders hit one receiver, the receiver knows best how much traffic is coming, so it tracks how many QPs are actively receiving and how many receive buffers are posted, then returns rate hints through ACKs. The slide's example is a 200 Gbps hint per sender.

4. Scale-Across: One NIC Runs Scale-Out and Cross-Campus Traffic Together

This is the most quantitative slide of the session. AMD's Balakrishnan Raman showed a single NIC handling two kinds of connections at once:

Item	Scale-out (same building)	Scale-across (cross-campus)
RTT	5 to 15 microseconds	Milliseconds, about 50 to 100x
Max congestion window	About 75 packets	3,500 to 7,000 packets
Max paths	About 12	About 30 to 50
Performance	Line rate	Line rate

Note: packets assume a 4K MTU; the slide diagram shows a 1.5 ms cross-campus RTT.

▲ The same AMD AI NIC handles both scale-out with 5–15 µs RTT and scale-across with millisecond-level RTT, the latter using 30–50 paths to reach line rate (Image source: Meta, AMD | 2026 OCP Global Summit presentation (p. 9))

The speaker's explanation: per-path windows are capped, so to fill a longer RTT the NIC dynamically opens more paths. A 1.5 ms RTT, at roughly 5 microseconds per kilometer of fiber, corresponds to about 150 km one way. Meta treats cross-campus training as a normal case for the same transport, not as a separate DCI solution.

5. AMD's Implementation: A Programmable AI NIC Turns the Protocol into Software

AMD is not a contract builder here but the first hardware implementation. The AMD programmable AI NIC on the slide has two P4DMA pipelines (Tx/Rx PDMA) and two P4NET parser/deparser stages around a shared packet buffer, with an 800 Gbps uplink and a PCIe Gen6 host interface. The slide does not name the part, but the 800G plus PCIe Gen6 combination matches AMD's announced Pensando Vulcano specifications.

Raman stressed that three functions are "implemented in software and tunable per deployment":

• Congestion control is a framework: beyond the built-in baseline algorithm and RCN, custom algorithms can be attached through programmable overrides.

• The receiver diagnoses reordering: it distinguishes reordering caused by spraying from reordering caused by drops, and only notifies the sender of real drops, avoiding needless retransmission.

• Multiplane distribution: traffic is spread evenly across all planes and the paths within them, and a failed plane fails over to the others.

We covered AMD's rack-scale strategy in AMD Q2 FY2026 Earnings (Helios); MetaRoCE effectively hands AMD's NIC a first-mover ticket on a hyperscaler protocol.

6. The Numbers: Small Messages 25% Faster, Large Messages Converge

Meta's Sandeep Nagaraja presented two data sets. The first is a lab 8x100G multiplane FPF network with eight disjoint planes:

• With 8 QPs, about 800 Gbps unidirectional and about 1,520 Gbps bidirectional, reaching the plateau from roughly 32KiB messages.

• With one plane removed, throughput fell from about 790 Gbps to about 670 Gbps, roughly one-eighth lower; the speaker said failures were detected in under a second and recovery caused no collateral damage to other paths.

▲ On an 8×100G multi-planar network, MetaRoCE reaches about 800 Gbps unidirectional and about 1,520 Gbps bidirectional; when a single plane fails, throughput drops by about one-eighth and returns to its original level after recovery (Image source: Meta | 2026 OCP Global Summit presentation (p. 11))

The second is 64-GPU collective completion time, with RoCEv2 as the baseline:

▲ 64-GPU collective completion time versus RoCEv2: All-Reduce is about 25% faster for small messages, narrowing to single digits at GiB sizes; All-to-All is 14–21% faster across most ranges, but several message sizes are actually slower (Image source: Simple Tech Trend original, data from Meta 2026 OCP Global Summit presentation (values read visually from charts, approximate))

The numbers need to be read separately. All-Reduce beats RoCEv2 across the entire range, by about 21% to 27% from 8B to 32KiB, shrinking to 3% to 11% at 1GiB and above. All-to-All is less tidy: 14% to 21% faster from 128B to 512KiB and about 29% faster at 64MiB, but about 19% slower at 0B, 16% slower at 1MiB, 10% slower at 32MiB, and 6% to 8% slower at 8GiB and above. The speakers credit the gains to replacing PFC with receiver-driven packet scheduling that avoids incast congestion at endpoints.

7. Risks and Counterarguments

• The test scale is small: 64 GPUs and an 8x100G lab network are three to four orders of magnitude short of the hundreds of thousands Meta itself describes; the real test is incast and multiplane behavior beyond 10,000 GPUs.

• All-to-All regresses in places: MoE expert parallelism depends heavily on All-to-All, and mid-size messages such as 1MiB and 32MiB got slower, with no explanation in the talk.

• The window numbers do not quite reconcile: the speaker said per-path windows are capped and more paths cover the BDP, yet on the slide the path count grows only about 3 to 4x while the total window grows about 50 to 90x; the specification is needed to confirm the details.

• The NIC carries more weight: a simpler switch means the NIC must hold per-path state, out-of-order placement and congestion decisions. The programmable pipeline reaches line rate, but area and power once hardened into an ASIC are not public.

• Standards fragmentation: UEC has already published a specification, and MetaRoCE adds another. If other hyperscalers do not follow, NIC vendors may have to support RoCEv2, UEC and MetaRoCE at once, raising validation costs.

8. What It Means for Taiwan's Supply Chain

• The bar for switches goes down, not up. MetaRoCE needs only ECN and ECMP, with no trimming or adaptive routing, which is good news for system makers building white-box switches on merchant silicon: Meta will not tie a transport upgrade to specific switch features.

• Value concentrates in the NIC, but no Taiwanese firm makes NIC silicon. AMD leads in programmable NICs, with Broadcom and NVIDIA as rivals; Taiwan's role is in NIC boards and server integration, so the metric to watch is 800G, PCIe Gen6 NIC shipments rather than the protocol itself.

• Multiplane multiplies connection counts. An 8x100G multiplane design splits one 800G NIC into eight independent 100G links to eight different planes, scaling up switch ports and fiber counts; that is incremental demand for fiber management and breakout cabling, though the talk gave no optical configuration and the scale remains to be confirmed.

• Scale-across is a long-term optical theme. A single transport handling millisecond RTTs implies cross-campus training becomes routine, and DCI bandwidth demand will grow with AI clusters; until Meta discloses actual distances and bandwidth, this is a direction rather than a number.

9. Conclusion

MetaRoCE matters not as yet another RDMA protocol, but because Meta has redrawn the division of labor in the AI network: the switch only forwards and marks, and every smart decision moves into the NIC. It fits Meta's long-standing playbook: keep hardware as generic as possible and put differentiation into software that can be open-sourced.

The numbers deserve an honest read. At 64 GPUs, the roughly 25% All-Reduce gain on small messages is real, while the All-to-All regressions remain unexplained. Meta seems to know this is not enough, which is why it open-sourced the specification, reference implementation and compliance suite, inviting NIC, DPU and switch vendors to validate it themselves. Three things to watch over the next 12 months: when a second NIC vendor announces a MetaRoCE implementation, when Meta publishes production data beyond 10,000 GPUs, and whether MetaRoCE and UEC converge or split.

Dumber switches, smarter NICs: that is Meta's bet on AI networking, and it is placing the bet by making every NIC vendor sit the exam on its specification.

This article is for technology and industry trend analysis only and does not constitute investment advice.