NVIDIA Now Certifies Cloud Clusters Before They Train AI Models

NVIDIA runs a formal validation program, called Exemplar Cloud, that checks whether a cloud provider’s rented GPU cluster actually performs like NVIDIA’s own reference hardware before customers start paying for it. Passing requires hitting at least 95 percent of the throughput NVIDIA’s reference architecture achieves on the same model, the same workload, and the same batch size, running on the same class of chip, whether that is an H100, a GB200 NVL72, or a GB300 NVL72 system. NVIDIA detailed the program’s diagnostic process in a July 30 developer blog post.

The finding that justifies the whole program is uncomfortable for buyers: two clusters built from identical chips routinely land 8 to 12 percent apart on training throughput, according to NVIDIA. Nothing about the silicon changed. The gap comes from a stack of configuration decisions in the kernel, the hypervisor, the BIOS, and NVIDIA’s own NCCL library (the software that coordinates communication between GPUs during training). Each setting alone costs a few percent. Stacked together, they can be large enough to fail certification even when every component on the invoice matches spec.

NVIDIA published four debugging cases from real partner clusters, and each one isolates a different failure layer worth knowing even for teams that will never seek certification.

The first involved a GB200 NVL72 cluster running inside a virtual machine, training a DeepSeek-V3 mixture-of-experts model. Iteration times ran 12 to 14 percent behind bare metal. A kernel-level trace found that nearly a quarter of CPU cycles were burned inside a single function that manages memory-address invalidation on NVIDIA’s Grace CPU under virtualization. Enabling a queue-virtualization feature called VCMDQ in the host kernel closed almost the entire gap. Dense models like Llama 3 barely noticed the same bug, since it only bites workloads that issue many small kernels per step.

The second case, an H100 cluster, ran Llama 3 70B pretraining 12 percent slower than reference for a mundane reason: the BIOS held idle CPU cores in a low-power state that is standard for latency-sensitive services but starves busy cores of the power budget they need to reach turbo clock speeds. A second issue stacked on top of it, with hypervisor housekeeping threads sharing physical cores with the data-loading process and causing repeated short stalls. Fixing the power state and separating the core assignments cut the gap to 3 percent.

The third case is the most counterintuitive. A GB300 NVL72 cluster with ConnectX-8 networking, rated for 1.6 terabits per second per node, showed a 31 percent gap that only appeared once training scaled to 512 GPUs. The cause was a single NCCL networking parameter left at its conservative default, which throttled how much of that bandwidth the collective communication operations could actually use. Raising it recovered most of the gap on this fabric, though NVIDIA is explicit that the same change can hurt performance on a different network or message size, so it has to be tested rather than applied blindly.

The fourth case shows how a fix can exist and still fail to apply: on a virtualized B200 cluster, an NCCL configuration file was set correctly on the host but never mounted inside the training container, so NCCL silently fell back to guessing the network layout. Throughput ran 13 to 53 percent below reference with no error message anywhere in the logs.

Another article in today’s issue makes the case that GPU utilization, not raw chip count, is the binding constraint on AI infrastructure. This post is NVIDIA’s own version of that argument, worth reading with the awareness that the company selling the chips also profits when more of them get bought to cover for the ones running below spec.

This is vendor material describing NVIDIA’s own certification process for NVIDIA hardware, and it should be read as such. For any team running large training clusters, the practical takeaway is to check BIOS power settings, container-visible NCCL configuration, and NUMA binding before assuming a slow cluster needs more GPUs rather than a fixed configuration.

NVIDIA published these findings on its developer blog on July 30, 2026.