Technical Paper Analysis | Cooling a 51.2T NPO Switch: An 835W ASIC Plus 16 Optical Engines - Air or Liquid?

At ITherm 2023, Celestica published a very "engineer's" paper: it took the thermal problem of a next-generation 51.2T Near-Packaged Optics (NPO) switch, ran Flotherm simulations for three approaches - integrated air cooling, separated air cooling and cold-plate liquid cooling - and delivered a straight answer. The core tension fits in one sentence: keeping the 835W ASIC in the middle below a 105°C Tj is not hard; the hard part is the 16 optical engines (OE) around it, each only 56W but locked to a 70°C case temperature spec. The conclusion is blunt: air cooling barely passes with no margin, while liquid cooling at 2 LPM gives both the ASIC and the OEs a safety margin, making it the right answer for the 51.2T generation.

1. Background: Why a Major ODM Takes Thermal Design So Seriously

First, who wrote this. The four authors, led by Yaoyin Fan, are from Celestica (a major Canadian electronics manufacturing services / ODM company), and the paper was presented at the 22nd IEEE ITherm in 2023. ITherm is a top venue in electronics cooling, so acceptance means its simulation boundary conditions and methods were vetted by peers.

Why would a contract manufacturer work so hard on thermals? Because in new architectures like NPO/CPO (co-packaged optics), thermal design itself is the key bottleneck that decides whether a switch can go into volume production - and that bottleneck usually lands on ODMs like Celestica, which handle full-system assembly and thermal design. In essence, this paper is Celestica flexing for customers: we have already done the thermal math on 51.2T NPO.

2. Why 51.2T Is a Thermal Nightmare

The root of the problem is not the optical modules or the PCB, but the switch ASIC's power curve, which has run out of control. The paper opens with Broadcom's ASIC roadmap to make the driver clear: from 196W for TH1 (3.2T) in 2013, up to 580W for TH4 (25.6T), and an estimated 800W+ for TH5 (51.2T). The 25.6T generation jumped about 60% over the previous one, and the paper uses a simulation value of 835W for the 51.2T generation.

Figure 1: Broadcom switch ASIC power roadmap, climbing from TH1 (196W) to TH5 (800+W). Source: Y. Fan et al., ITherm 2023 - Figure 1

NPO's answer is to move multiple optical engines right next to the ASIC, shortening electrical paths and eliminating a large share of SerDes power, so total power is much lower than in traditional architectures.

Figure 2: NPO architecture, with the switch ASIC at the center surrounded by optical engines. Source: Y. Fan et al., ITherm 2023 - Figure 2

But what is saved is total system power; what you get in exchange is a serious hotspot density problem. Simulated component specs (Table 1): ASIC package 87×75×4.62mm, die 31.37×24.74×0.84mm, 835W, Tj spec 105°C; OE chip 52×22.5×7.6mm, 56W each, case temperature (Tc) spec only 70°C. Two key gaps: the ASIC draws nearly 15x the power of an OE, yet the OE's temperature spec is 35°C stricter. This is the source of every design trade-off in the paper.

3. Two NPO Cooling Paths the Industry Has Already Taken

The first is Meta. At an OCP tech talk in May 2022, Meta proposed pairing a 51T ASIC with surrounding NPO ports in a 4RU chassis, using a single integrated heat sink covering both the ASIC and 16 optical modules.

Figure 3: Exploded view of Meta's NPO design, using a single integrated heat sink to cool the ASIC and 16 optical modules. Source: Y. Fan et al., ITherm 2023 - Figure 3

The second is Ragile. At the OCP Global Summit in September 2022, Ragile showed working 51.2T and 25.6T NPO switches, both using cold-plate liquid cooling.

Figure 4: Ragile's 51.2T NPO switch hardware shown at the 2022 OCP Global Summit. Source: Y. Fan et al., ITherm 2023 - Figure 4

Meta went with integrated air cooling and Ragile with liquid cooling; this paper's contribution is to pit all three approaches against each other fairly under the same boundary conditions.

4. The Physics Behind Three Cooling Weapons

Heat pipes (HP) and vapor chambers (VC) are veterans of ASIC cooling, moving heat through phase change of an internal working fluid. The VC base sits on the hot chip and spreads heat out to the fins; heat pipes soldered to the base then carry heat from the VC to the tops of the fins, improving fin efficiency and lowering spreading resistance.

Figure 5: How heat pipe and vapor chamber heat sinks work, transferring heat through working-fluid phase change. Source: Y. Fan et al., ITherm 2023 - Figure 5

The 3D vapor chamber (3D VC) is the advanced version - effectively a round heat pipe combined with a flat vapor chamber, with no contact resistance between them, enabling both planar and axial heat transfer for true three-dimensional heat spreading.

Figure 6: How a 3D vapor chamber works, combining planar and axial heat transfer for three-dimensional heat spreading. Source: Y. Fan et al., ITherm 2023 - Figure 6

The third is cold-plate liquid cooling. The paper chose the most mature and cost-controlled option, single-phase cold-plate cooling: coolant (water) enters the cold plate, is heated by chip heat flux as it flows through the channels, and carries the heat out through the outlet. Cold plate design quality directly determines thermal performance.

Figure 7: Cold plate cross-section, with coolant carrying chip heat away through internal channels. Source: Y. Fan et al., ITherm 2023 - Figure 7

5. Air Cooling Option 1: Integrated Heat Sink (ASIC and OEs Share One)

Boundary conditions: 4U chassis, air-cooling simulation domain 500×326×118.5mm, maximum heat sink envelope 320×230×103mm; 35°C inlet air, sea level.

Figure 8: Air-cooling simulation domain, 35°C inlet air, 118.8mm chassis height. Source: Y. Fan et al., ITherm 2023 - Figure 8

Integrated version 1, VC+HP: overall 320×200×92mm, 1.6mm fin pitch, 0.3mm fin thickness, aluminum fins, VC base plus 10 heat pipes (Table 2).

Figure 9: VC+HP integrated heat sink, with 10 heat pipes to improve fin efficiency. Source: Y. Fan et al., ITherm 2023 - Figure 9

Version 2 is a 3D VC integrated heat sink with the same dimensions and fin parameters (Table 3); the difference is that 3D VC replaces VC+HP.

Figure 10: 3D VC integrated heat sink. Source: Y. Fan et al., ITherm 2023 - Figure 10

The fatal flaw of the integrated design: when OEs share a heat sink with the ASIC, OE temperature is inevitably dragged up by the ASIC - the OE's 70°C case spec is far below the ASIC's 105°C, while the ASIC's power is much higher. This leads directly to the next option.

6. Air Cooling Option 2: Separated Heat Sinks (Rescuing the OEs from the ASIC)

The logic of the separated design: give the ASIC and the OEs their own heat sinks to physically isolate heat flows. The paper designs five separate heat sinks - one for the front OEs, two for the side OEs, one for the rear OEs and one for the ASIC.

Figure 11: Separated heat sink layout, with the ASIC and OEs on all four sides cooled independently. Source: Y. Fan et al., ITherm 2023 - Figure 11

Two clever touches (Table 4, Table 5): the ASIC heat sink is 120×218×81mm with VC+10 HPs; to push more airflow to the central ASIC, the front OE heat sink deliberately leaves out fins in the middle to form an air channel; the side OEs use 6 heat pipes to carry heat to parallel fins, freeing space for the ASIC. The OEs are rescued, but the fin area available to the ASIC is squeezed.

7. Liquid Cooling: A Dual-Loop Cold Plate

Configuration (Table 6): 1 ASIC (836W) + 16 OEs (56W each), for a total liquid-cooled heat load of 1732W; open loop, single phase; design flow rates of 1.6/2/3 LPM; water as the working fluid; FWS maximum supply temperature 32°C, TCS supply temperature 40°C.

Key difficulty: the ASIC draws 835W from a die of only 31.37×24.74mm, pushing heat flux to 108 W/cm² with very high spreading resistance. The fix is a VC pedestal under the cold plate to cut spreading resistance, plus a large VC base plate at the bottom of the cold plate to spread heat evenly into the coolant.

Figure 12: Cold-plate liquid cooling assembly, with a VC pedestal under the ASIC to reduce spreading resistance. Source: Y. Fan et al., ITherm 2023 - Figure 12

Based on the NPO chip layout, a two-inlet, two-outlet cold plate was designed (Table 7: ASIC channel 80×62×5.5mm, 1.1mm fin pitch; single OE channel 32×20×5.5mm, 1.3mm fin pitch; 0.2mm fin thickness, copper fins).

Figure 13: Cold plate internal channel and coolant flow design. Source: Y. Fan et al., ITherm 2023 - Figure 13

8. The Underrated Factor: TIM Selection and Clamping Force

When ASIC heat flux reaches 108 W/cm², choosing the wrong thermal interface material (TIM) makes contact resistance large enough to guarantee the chip overheats. For the ASIC, the phase change material (PCM) Laird TPCM 7900 was selected. The OE side is where precise calculation matters, because TIM thickness is heavily affected by assembly tolerances. The paper uses a one-dimensional assembly tolerance stack-up analysis (TA).

Figure 14: Structural stack-up covered by the TA tolerance analysis. Source: Y. Fan et al., ITherm 2023 - Figure 14

Figure 15: Definitions of key dimensions A-F in the TA tolerance analysis. Source: Y. Fan et al., ITherm 2023 - Figure 15

The TA results (Tables 8-10) put compressed OE TIM thickness at 0.43-1.12mm, so a thermal pad at least 1.5mm thick is needed. Four pads compared (Tables 11, 12): Fujipoly GR130A (13 W/mK) needs >300.7 LBF clamping force and cannot be assembled; PG130A (13) 169.5 LBF; Laird SF10 (10) 28.6 LBF; Lipoly Twork9000 (20) 21.8 LBF. Conclusion: Twork9000 has higher conductivity (20 vs 10) yet needs less clamping force (21.8 vs 28.6 LBF), making it the best choice. Thermal conductivity is not the only metric - clamping force is the gate to volume production.

9. Simulation Showdown of the Three Options: Let the Numbers Talk

The tool is the commercial CFD software Flotherm. Two integrated versions (Fig 16, 17): VC+HP gives ASIC Tj 93.0°C and hottest OE 70.2°C; 3D VC gives ASIC Tj 91.3°C and hottest OE 71.5°C. 3D VC lowers the ASIC but makes the OEs hotter (the number of heat pipes in the OE area is limited), so heat pipe layout optimization is key.

Figure 16: ASIC and OE temperature distribution with integrated heat sinks ((a) HP-VC; (b) 3D VC). Source: Y. Fan et al., ITherm 2023 - Figure 16

Figure 17: Fin temperature profile along the airflow direction for the integrated heat sinks. Source: Y. Fan et al., ITherm 2023 - Figure 17

Separated (Fig 18): ASIC Tj 95.1°C (2.1°C worse), but the hottest OE drops to 68°C (2.2°C lower) - everything passes with margin. The front OEs run coolest (they get un-preheated inlet air), and the side and rear OEs are evenly distributed.

Figure 18: ASIC and OE temperature distribution with separated heat sinks; all OEs stay within the 70°C spec. Source: Y. Fan et al., ITherm 2023 - Figure 18

Liquid cooling (Fig 19-21) is the best of the three. At 1.6 LPM the OE reaches 73.6°C, 3.6°C over spec; at 2 LPM ASIC Tj is 84.9°C and OE 66.5°C, both with margin, so a design flow rate of at least 2 LPM is recommended.

Figure 19: ASIC and OE temperatures with cold-plate liquid cooling at 1.6/2/3 LPM. Source: Y. Fan et al., ITherm 2023 - Figure 19

At 2 LPM (Fig 20, 21), the VC base plate temperature is uniform. OEs heat up progressively along the flow direction - OE 8# is about 8°C hotter than OE 1#, because the coolant warms up along the way and downstream chips naturally run hotter. This is the common fate of series liquid cooling.

Figure 20: ASIC and OE temperature distribution with cold-plate liquid cooling at 2 LPM. Source: Y. Fan et al., ITherm 2023 - Figure 20

Figure 21: OE temperature rises along the coolant flow; OE 8# runs about 8°C hotter than OE 1#. Source: Y. Fan et al., ITherm 2023 - Figure 21

10. Don't Forget the Mechanics: Clamping Force and Deformation Risk

The integrated design presses the ASIC and OEs under one heat sink, so tolerance stack-up easily creates stress concentration. With Twork9000 as the OE TIM, maximum clamping force per OE is 21.8 LBF; the ASIC's spec maximum is 82.3 LBF, and with a 0.8 safety factor the design clamping force is 65.8 LBF. Reinforcement: a support bracket on the back of the PCB, and stainless steel stiffeners on the back of the VC base and the middle of the fins.

Figure 22: Stainless steel stiffeners for the VC base and fins. Source: Y. Fan et al., ITherm 2023 - Figure 22

Deformation simulation (Fig 23): maximum deformation around the ASIC is 0.574mm, and with the bracket the fins are well protected. Safety factor simulation (Fig 24): with the bracket, the structure reaches a safety factor >1.

Figure 23: Deformation simulation; maximum deformation around the ASIC is 0.574mm. Source: Y. Fan et al., ITherm 2023 - Figure 23

Figure 24: Safety factor simulation; with stiffeners the structure reaches a safety factor >1. Source: Y. Fan et al., ITherm 2023 - Figure 24

11. STT View: The Real Industry Signal in This Paper

First, it completes the proof for the whole industry on the question "can NPO be cooled?" Separated air cooling barely passes (OE 68°C), and liquid cooling at 2 LPM passes comfortably (OE 66.5°C), so the threshold has been verifiably lowered - a shot in the arm for system makers still weighing when to adopt NPO/CPO.

Second, the OE, not the ASIC, is the thermal bottleneck. What really blocks the design is the 35°C spec gap - the OE's 70°C case temperature ceiling is the center around which every design trade-off revolves. This is a clear signal for optical engine suppliers (players making ELSFP / optical engine modules): your module's thermal design and package Tc performance will directly decide whether customers can use air cooling or are forced into liquid cooling. OE thermal friendliness is becoming a competitive edge.

Third, liquid cooling is the trend, but air cooling is not out yet. The paper states plainly that VC+HP is better suited than 3D VC, because OE cooling is the bottleneck and 3D VC is not yet as mature as VC+HP. In the 51.2T generation, maturity and supply chain availability sometimes matter more than on-paper thermal performance. A full shift to liquid cooling may have to wait for the 102.4T generation.

In one sentence: in the 51.2T NPO thermal battle, the deciding factor is not the fiercest chip - the ASIC - but the 16 most delicate optical engines.

Summary

Through a fair head-to-head of three cooling approaches, Celestica's ITherm 2023 paper sets a credible reference baseline for 51.2T NPO switch thermal design. Three conclusions: first, cold-plate liquid cooling at 2 LPM leaves margin for both the ASIC and the OEs and is the most robust solution for the 51.2T generation; second, if you insist on air cooling, separated beats integrated and VC+HP beats 3D VC - the key is isolating the OEs from the ASIC's heat flow; third, TIM selection cannot be based on conductivity alone - clamping force is the gate to volume production. For the optical engine supply chain, a module's Tc thermal performance is turning from a line on a datasheet into a competitive factor that decides the customer's cooling architecture.

References

• Y. Fan, M. Luo, C. Liu, S. Zhang, "Thermal solution study of 51.2T Near-packaged optics switch," 2023 22nd IEEE ITherm, 2023. DOI: 10.1109/ITHERM55368.2023.10177553