Fine-tuning a Mixture-of-Experts model without frontier-scale GPU budgets has been, until now, a problem of pure hardware arithmetic. NVIDIA published benchmarks on June 15, 2026, showing that its open-source NeMo AutoModel library can narrow that gap significantly, at least on its own hardware and by its own measurement.
The library sits on top of Hugging Face Transformers v5, which shipped first-class MoE support earlier this year. NeMo AutoModel subclasses Transformers’ standard AutoModelForCausalLM, meaning the code change required to adopt it is a single import line swap. According to NVIDIA’s technical blog on Hugging Face, that single change delivers 3.4-3.7x higher training throughput and 29-32% lower peak GPU memory versus the best Transformers v5 configuration, on 30-billion-parameter MoE models running on a single 8xH100 node.
Three mechanisms drive the gains. First, Expert Parallelism shards expert weights across GPUs rather than replicating them, so an 8-GPU node holds only one-eighth of the expert parameters per device. For Qwen3-30B-A3B, this cuts peak memory from 68.2 GiB to 48.1 GiB per GPU. For Nemotron 3 Nano 30B A3B, it falls from 62.1 GiB to 42.5 GiB. Second, DeepEP (an open kernel library originally released by DeepSeek) fuses token routing into GPU kernels, overlapping communication with expert computation instead of running them sequentially. Third, NVIDIA’s TransformerEngine provides fused attention and linear-layer kernels that accelerate every layer, not just MoE-specific ones. According to NVIDIA, this kernel stack reduced cost per iteration by 47% on the full DeepSeek V3 671B model versus all-gather baselines.
The scale story is harder to verify independently. At 550B parameters (NVIDIA’s own Nemotron 3 Ultra), Transformers v5 runs out of memory entirely on 128 H100 GPUs, so there is no apples-to-apples comparison. NeMo AutoModel completes the full fine-tune at EP=64 across 16 nodes, achieving 815 tokens per second per GPU. The absence of a v5 baseline at that scale is not cherry-picking, but it does mean the throughput figures stand alone rather than as a ratio.
The methodology note in the blog is worth reading carefully. The single-node 30B benchmarks use a balanced routing gate, which distributes tokens uniformly across experts and reflects a well-trained model’s steady-state. Native Transformers v4 and v5 run their actual routers on dummy tokens. NVIDIA argues this reflects the real operating point that a converged model converges to. That framing is reasonable. It also happens to produce the cleanest possible number for NeMo AutoModel, since balanced routing eliminates straggler noise that would otherwise reduce utilization. Teams evaluating this on their own workloads should re-run on real data before committing to a training stack change.
The practical value here is not primarily for teams running 550B models. Those teams already have the infrastructure. The value is for smaller shops that want to fine-tune a capable open MoE, such as Qwen3 or DeepSeek V3, on a single rented node. MoE architectures now dominate the frontier: Qwen3, DeepSeek V3, Mixtral, and NVIDIA’s own Nemotron family all use sparse expert routing. The parameter counts look modest in isolation, but the expert weight footprint scales with total parameters, not active ones, which is what fills GPU memory during training.
NeMo AutoModel checkpoints write standard Hugging Face safetensors format. That means a model fine-tuned through this stack can load directly into vLLM or SGLang for inference, without a conversion step. For teams already operating a Hugging Face-centered workflow, that portability lowers the barrier to adoption.
Teams currently evaluating open MoE fine-tuning should benchmark NeMo AutoModel against their target model and real training data before the end of Q3, when fine-tuned domain-specific MoEs will face meaningful competition from quantized dense models that fit on fewer GPUs with simpler infrastructure.
Detailed in NVIDIA’s technical blog on Hugging Face, published June 15, 2026, authored by Adil Asif, Alexandros Koumparoulis, Wenwen Gao, Sylendran Arunagiri, David Messina, and Bernard Nguyen.
