This is the 4

Jan 16, 2024

At ISC 2023, we saw a 4-node dual-socket blade. Dubbed the HPE Cray EX420, this blade is designed for CPU compute applications instead of GPU acceleration. The CPUs are placed into a chassis, along with Slingshot high-speed networking and more all in a liquid-cooled chassis.

At ISC 2023, the blade was displayed in a large case along with two GPU compute blades we have seen before. What was interesting was the placard for the display.

HPE says that this was for "Intel's next-Gen Sapphire Rapids processor." What is a bit strange about this placard is that the Sapphire Rapids processors are DDR5 RDIMM, not DDR4 processors.

Here is a look at the blade itself. One can see the four dual-socket nodes in this blade.

Each node has eight DIMM slots per CPU and we were told that typically memory capacities were not extreme with 16GB, 32GB, and 64GB being the common capacities. That puts the 16x DIMM slot node at a maximum of around 1TB of memory.

The entire blade is liquid-cooled, so there are no fans onboard. That includes not just the CPUs, but also the memory, interconnects, and everything else in the system.

Networking is provided by the HPE Slingshot interconnect. The slingshot cards are also liquid-cooled in the chassis.

Each node can optionally come with M.2 storage.

Something that we did not catch earlier, but as we were looking at photos did not make sense, aside from the DDR4 and Sapphire Rapids spec, was the CPU cooler mounting. Normally Intel CPU sockets in the Ice Lake and Sapphire Rapids have heatsink mounting points at the corners of the socket, as we can see from our recent Supermicro SYS-111C-NR 1U Intel Xeon Server Review.

When we looked at the photos of the HPE Cray EX420 (and validated with the service manual), the unit on display at SC23 was not an Intel Xeon Sapphire Rapids server. Instead, it was an AMD EPYC 7002 Rome system. We can see the telltale green carrier tab and AMD EPYC 7001-7002-7003 SP3 cooler mounting pattern.

That made this showcase perhaps more interesting than when we first saw it.

The HPE Cray EX chassis has so much liquid cooling that it can be hard to tell what is underneath. Since we had the "Sapphire Rapids" launch months ago, this was not the case where it was an unreleased CPU. Instead, our best guess is that it is the blade HPE had on hand in Europe and so it put an Intel Xeon Sapphire Rapids placard on an AMD EPYC Rome system.

Regardless of the CPU, this was still an interesting node to see. The HPE Cray platforms are the top-end supercomputer platforms these days, especially the accelerated versions. It was great to also see a CPU-only blade. Normally when we review four dual-socket systems they are in the form factor of 2U 4-node servers where a huge amount of power is consumed by the fans pushing air through the dense chassis. This liquid-cooled design is a few steps beyond the 2U 4-node setup we did at CoolIT with AMD EPYC 7003 "Milan" CPUs.

In the How Liquid Cooling is Prototyped and Tested in the CoolIT Liquid Lab Tour you can see how some of these HPE Cray EX nodes are liquid cooled including one that powers Frontier, the world's #1 supercomputer: