NVIDIA Breaks 1,000 TPS Barrier with Blackwell GPUs and Meta’s Llama 4 Maverick for Record Token Speeds

NVIDIA has made a significant breakthrough in artificial intelligence (AI) performance with the introduction of its Blackwell architecture. This innovation is largely attributed to a series of strategic optimizations and enhanced hardware capabilities.

Advancements in Blackwell: Elevating AI Performance for Large-Scale Language Models

Continuously pushing the boundaries of AI, NVIDIA has made remarkable progress with their Blackwell technology. In a recent blog post, the company announced that they have reached an impressive milestone of 1, 000 tokens per second (TP/S) using a single DGX B200 node equipped with eight NVIDIA Blackwell GPUs. This achievement was accomplished while working with Meta’s substantial 400-billion-parameter Llama 4 Maverick model, showcasing the profound impact of NVIDIA’s AI ecosystem on the industry.

With this advanced configuration, NVIDIA’s Blackwell servers can deliver up to an astonishing 72, 000 TP/s. As highlighted by CEO Jensen Huang during his Computex keynote, organizations are now more motivated than ever to showcase their AI advancements, particularly in terms of token output rates. This trend indicates NVIDIA’s strong commitment to enhancing this specific aspect of AI development.

Achieving such groundbreaking speed involves significant software optimizations, notably through TensorRT-LLM and an innovative speculative decoding model, resulting in a fourfold acceleration in performance. NVIDIA’s team delves into various elements that contributed to fine-tuning Blackwell for extensive large language models (LLMs).A pivotal innovation is the use of speculative decoding, a method that employs a nimble “draft” model to forecast several tokens ahead, while the principal (larger) model concurrently validates these predictions.

Speculative decoding is a popular technique used to accelerate the inference speed of LLMs without compromising the quality of the generated text. It achieves this goal by having a smaller, faster “draft” model predict a sequence of speculative tokens, which are then verified in parallel by the larger “target” LLM.

The speed-up comes from generating potentially multiple tokens in one target model iteration at the cost of extra draft model overhead.

– NVIDIA

Moreover, NVIDIA has implemented the EAGLE3-based architecture, a software-focused framework designed specifically to enhance the inference processes for large language models, rather than relying purely on GPU hardware advancements. With these developments, NVIDIA not only asserts its leadership position within the AI domain but also positions Blackwell as an optimized solution for prominent LLMs like Llama 4 Maverick. This milestone represents a pivotal step towards facilitating more rapid and seamless AI interactions in the future.

Source & Images