Super Micro Computer’s NVIDIA HGX B200 Achieves Over 3X Tokens Per Second for Llama2-70b and Llama3.1-405b Benchmarks Compared to H200 8-GPU Systems

Please note that this content does not constitute investment advice. The author holds no positions in the stocks discussed herein.

Super Micro Computer’s Resilience in the Tech Market

Despite the turbulence affecting the technology sector, Super Micro Computer (NASDAQ: SMCI) has managed to surge 6% this year. This performance is noteworthy, especially as major companies face challenges in the current economic landscape. Recently, Goldman Sachs recognized SMCI as the “best performing stock”within the hardware category. In a further display of innovation, Super Micro has announced a significant advancement in the AI capabilities of its systems powered by NVIDIA’s B200 GPU.

AI Performance Leadership Announcement

Super Micro is now heralding its “4U liquid-cooled and 10U air-cooled systems”as outperformers, claiming they generate over three times as many tokens per second (Token/s) for Llama2-70B and Llama3.1-405B benchmarks when compared to H200 8-GPU systems. This leap in performance is a testament to the company’s commitment to advancing AI technology.

“Within the operating margin, the Supermicro air-cooled B200 system exhibited the same level of performance as the liquid-cooled B200 system.”

Innovative Cooling Technologies

The latest NVIDIA HGX B200 systems from Super Micro are designed with advanced cooling technologies, including newly engineered cold plates and a 250kW coolant distribution unit. This innovative design employs vertical coolant distribution manifolds, maximizing rack space efficiency. As a result, it can accommodate **eight systems** with a total of **64 NVIDIA Blackwell GPUs** in a **42U rack**, or even **12 systems** with **96 NVIDIA Blackwell GPUs** in a **52U rack**.

“The new air-cooled 10U NVIDIA HGX B200 system features a redesigned chassis with expanded thermal headroom to accommodate eight 1000W TDP Blackwell GPUs. Up to 4 of the new 10U air-cooled systems can be installed and fully integrated in a rack, the same density as the previous generation, while providing up to 15x inference and 3x training performance.”

Understanding Server Height Measurement

For those unfamiliar, the measurement “U”refers to server height, where 1U is equivalent to 1.75 inches.

Apple’s Strategic Shift

In a recent development, Loop Capital revealed that Apple is making its entrance into the “large server cluster Gen AI market.”This strategic pivot comes in response to recent challenges, particularly surrounding Siri, nudging the company towards NVIDIA’s commercial GPUs.

“AAPL [Apple] is in the process of placing orders for ~$1.0B of GB300NVL72’s (or ~250 servers at $3.7M – $4.0M each) comprised of both SMCI [Super Micro Computer] & DELL.”

This substantial order marks a significant success for both Super Micro Computer and Dell, who are currently leading suppliers in the burgeoning AI server rack market.

Source & Images