OCP Global Summit 2026 | SemiAnalysis | Dylan Patel | From Silicon to Services

1. Why SemiAnalysis Is the One Talking "Silicon to Services"

SemiAnalysis now has about 110 people across 15 countries (spoken), with backgrounds spanning equipment vendors, fab R&D, chip design, hyperscaler deployment and power-plant construction. Its slide lists what it builds: 14 institutional models, 2 open benchmarks (InferenceX, AgentX), 1 cloud rating system (ClusterMAX) and 1 free GPU price index. It also runs a reverse-engineering lab in Oregon with more than $10 million of equipment (spoken), most recently tearing down two 2 nm-class processes, Intel 18A and Samsung SF2.

We covered its view on copper versus optics for scale-up in OCP APAC Summit 2026 | SemiAnalysis | Scale Up Sophistry. The Global Summit talk has a completely different focus: not which medium carries the interconnect, but how many tokens the whole data center ultimately delivers.

Every debate about interconnect, memory and power eventually has to be converted to one denominator: how many tokens each MW can sell.

2. Breaking Down Demand: Agentic Is Now the Mainstream Traffic

Patel's first point is that the old benchmark habit of fixed input/output ratios is obsolete. Real agentic coding alternates model turns with tool calls; during a tool call the GPU generates no tokens, so other users must be merged in through continuous batching to keep the hardware busy.

SemiAnalysis built AgentX for exactly this:

• 393 anonymized sessions replayed, selected from 8,300 real Claude Code sessions (3.4 million requests, more than $3 million of spend)

• Input length P50/P90/P99 of 88K/272K/675K tokens, with a maximum context of 1M

• 44% of sessions launch subagents

The demand evidence comes from OpenAI's own data: Codex's share of enterprise output tokens accelerated after the Codex macOS app launched in February 2026 and reached 64% in June, leaving ChatGPT with 36%. Patel added verbally that the share is "even higher" now.

▲ Agentic (Codex) share of OpenAI enterprise output tokens, rising from near zero in August 2025 to 64% in June 2026 (Image source: SemiAnalysis | 2026 OCP Global Summit presentation (slide 8))

The hardware implication is direct: longer context means a bigger KV cache, and memory capacity and bandwidth become the bottleneck. One number on the slides backs this up: enabling CPU KV offload in vLLM raises output throughput by 81.7%.

3. Breaking Down Supply: Rubin's 7.2x Only Appears at 150 tok/s

This is the most important table of the talk. The setup is DeepSeek V4 Pro (1.6T parameters), the AgentX workload and FP4; the unit is million tok/s per all-in utility MW. Rubin is still on pre-release software.

P90 interactivity (tok/s/user)	Best GB300 (M tok/s/MW)	Vera Rubin NVL72 (M tok/s/MW)	Rubin / GB300
75	44.14	64.00	1.45x
100	28.47	59.38	2.09x
125	12.01	51.55	4.29x
150	5.14	36.98	7.19x
170	3.92	21.78	5.56x
200	2.73	7.43	2.72x

▲ Vera Rubin NVL72's tokens-per-MW multiple over the best GB300, peaking at 7.19x at 150 tok/s/user and converging at both ends (Image source: Simple Tech Trend original; data from SemiAnalysis 2026 OCP Global Summit presentation)

Three things to note here:

• Each chip draws more power. In SemiAnalysis's model, all-in power per chip is 3.3 kW for VR200 and 2.12 kW for GB300, about 56% higher (STT calculation). Under per-MW accounting this is not a weakness, because output rises faster.

• The multiple is not a single number. The community used to expect "2x performance per generation at 50% higher cost"; Patel says that rule of thumb no longer holds under model, hardware and software co-optimization, but the multiple depends heavily on the operating point.

• Commercial services operate at 75–150 tok/s/user (spoken). That means interactive services such as OpenAI and Anthropic get the highest return from moving to Rubin, while operators running slow batch inference have far less urgency to upgrade.

In a power-capped data center, this table is the upgrade schedule.

4. Breaking Down Cost: Same Model, Chip Rankings Flip

InferenceX reruns every day on GitHub Actions, covering 13 accelerator SKUs and more than 10 serving stacks. SemiAnalysis uses its own TCO model to convert each chip into an hourly cost (owning at hyperscaler scale): H100 $1.17, B200 $1.73, B300 $2.26, GB300 $2.31, MI355X $1.50; TPUv7 is $1.03 on Google's internal TCO.

With these costs applied, the conclusion changes with the operating point:

• NVIDIA leads most of the range: on Qwen3.5 at 90 tok/s/user, NVIDIA's lead over AMD exceeds 20x; on GLM 5.3 at 150 tok/s/user, the cost-efficiency gap reaches up to 5x. At low speeds the gap is much smaller.

• AMD takes the far right end: MI355X with vLLM prefill and TileRT decode reaches a peak P90 interactivity of about 470 tok/s/user, a speed none of the engines tested on NVIDIA can reach. The slide also notes higher TTFT, so peak streaming speed alone does not establish a cost-efficiency win.

• TPU is cheapest at 100 tok/s/user: on Qwen3.5 397B, FP8, 8K/1K, TPUv7 costs $0.181 per million tokens on external TCO, about 18% below B200 and 34% below B300 (STT calculation). The slides show B200 and GB200 leading at low latency, with TPUv7 taking the lead as latency relaxes. This is still a preview; FP4, disaggregated prefill/decode, MTP and AgentX are not yet included.

▲ Cost per million tokens at 100 tok/s/user: TPUv7 at $0.181, below B200 at $0.222 and B300 at $0.276 (Image source: Simple Tech Trend original; data from SemiAnalysis 2026 OCP Global Summit presentation)

▲ AMD MI355X with TileRT reaches about 470 tok/s/user on GLM 5.2/5.3, beyond the range of the GB300 curve (Image source: SemiAnalysis | 2026 OCP Global Summit presentation (slide 11))

One more signal from the Q&A: SemiAnalysis has tested OpenAI's in-house Jalapeno chip, and Patel says it beats GB300 across the entire curve on open models, though it will not ramp until next year (spoken).

5. Breaking Down Services: Hundreds Sell Tokens, Few Can Supply at Scale

Patel was specific about the inference market: renting GB300 from a cloud and running open models on open frameworks can yield roughly 70% gross margin at high utilization (spoken). As a result, more than 100 companies now sell tokens.

But supply is highly concentrated: most players can only get dozens of racks, hundreds to thousands of GPUs; supplying at scale is the tier of Amazon Bedrock (hundreds of thousands of Trainium chips and GPUs) and Fireworks (tens of thousands of GPUs) (spoken). ClusterMAX data echoes this: of 323 cloud providers tracked and 77 actually reviewed, only 19 earned a medallion, about a quarter of those reviewed.

Another challenge is inconsistent quality. The same open model can produce very different coding eval results across providers, because some providers make optimizations that sacrifice accuracy to cut cost. Patel noted that token routing platforms are emerging, but he did not call whether the market will consolidate toward large operators or be brokered by a routing layer across small ones.

SemiAnalysis laid out its own watch list for the next 6 to 12 months:

• Rubin NVL72 moving from preview to official results

• AMD MI455X UALoE72 rack testing, which AMD has agreed to open up in its own data centers (spoken)

• TPUv8i (native FP4) to be added in about 6 months (spoken)

• SambaNova SN50 and Amazon Trainium3 are planned, with no results yet

For background on AMD's rack-scale product, see Earnings Highlights: AMD | Q2 FY2026 (Helios).

6. Risks and Counterarguments

• Rubin numbers are a preview: pre-release software and modeled all-in power; until official results arrive, 7.19x should be treated as an upper bound.

• TCO assumptions drive the ranking: hourly costs assume owning at hyperscaler scale, and TPU has both internal and external TCO figures. Rental pricing or small deployments may change the conclusion.

• The 64% covers only OpenAI enterprise: it is one company's output-token share for one customer segment and cannot be extrapolated to the whole inference market.

• Spoken numbers are unverified: the 70% margin, Jalapeno winning across the curve, and the scale of Bedrock and Fireworks have no source on the slides.

• The benchmark has a sponsor ecosystem: InferenceX receives donated hardware and code from chip vendors, model labs and hyperscalers. Open methodology and CI are strengths, but testing priorities are still shaped by partnerships.

7. What It Means for Taiwan's Supply Chain

• Foundry is unaffected by who wins the benchmark. The competing NVIDIA, AMD and Google TPU accelerators on these charts are all made at TSMC, so share shifts among the three have limited impact on TSMC's leading-edge process and advanced packaging demand. The real variable is total volume, and the growth of agentic traffic is pushing that total up.

• Per-chip power rises from 2.12 kW to 3.3 kW, meaning each accelerator needs more power conversion and cooling capacity. For Taiwan's rack assembly, power and liquid-cooling supply chain, content per chip rises with power; but the talk named no Taiwanese companies, so the benefit to any individual company depends on its own customers and product mix.

• Optical communications were not mentioned this time, and that is the point. When data centers price output in tokens per MW, the pJ/bit claims of optical modules and CPO only persuade once they are converted into "how many more tokens the saved energy produces." For an interconnect energy baseline, see OIF Draws an Official Map for AI Interconnect.

• Long context raises memory demand. CPU KV offload delivers an 81.7% throughput gain, which means server-side memory capacity becomes part of inference performance, a direction worth tracking for server board and memory module makers.

8. Conclusion

The significance of this talk is that SemiAnalysis has changed how AI infrastructure is scored, from chip specs to service output: how many tokens each MW and each dollar delivers at a given interactivity target. By that measure, Rubin's value concentrates around 150 tok/s/user, AMD and TPU each win in specific ranges, and the software stack matters as much as the chip.

The timing needs to be separated. Demand has already arrived: agentic is 64% of OpenAI enterprise output, with a median context of 88K. Supply is still being validated: Rubin official results, the MI455X rack and TPUv8i are expected over the next 6 to 12 months. Three things to watch next: whether official Rubin numbers hold the 7x gap at 150 tok/s, how AMD and TPU perform on AgentX long-context tests, and whether the inference service market consolidates toward large operators.

Once the unit of account becomes tokens per MW, every hardware debate gets repriced, including optical interconnect.

This article is for technology and industry trend analysis only and does not constitute investment advice.