NVIDIA researchers have published a method that shrinks a language model until it fits a specific GPU, rather than a specific parameter count, and used it to cut Nemotron 3.5 Super from 120 billion to 74 billion parameters. The preprint, AutoCompress, appeared on arXiv on 5 November 2026. The 74B version serves at 128K context on a single B200 with 178 GB of memory in use, under the card’s 180 GB limit.

Pruning means removing parts of a trained model (whole layers, attention heads, groups of experts, slices of the hidden width) and then retraining the smaller result to recover what it lost. The usual workflow starts from a target such as “make it 8 billion parameters.” The authors argue that number is a poor stand-in for what a deployment needs. In one example from the paper, two architectures with identical parameter counts differ by 35 percent in latency on an RTX 5070, because a narrow, deep design runs slower than a wide, shallow one. In a second, a pair of designs deliver matching H100 throughput even though one carries 2.2 times as many parameters.

AutoCompress instead takes the hardware, the workload, and a limit on latency, throughput, or memory, then searches for the architecture that keeps the most quality inside that limit. It measures how long each layer type actually takes inside the vLLM serving engine and uses those timings to predict the cost of any candidate. Each layer is allowed to end up a different size, which earlier uniform approaches such as NVIDIA’s own Minitron did not allow.

The paper tests three cases. For a game character on an RTX 5070, the 12B Nemotron Nano parent took 1,147 ms and 8.76 GiB on the target workload. The pruned model measured 739.7 ms and 3.93 GiB against a 750 ms and 4 GiB budget. For a local coding agent on an RTX 5080, a 31.6B parent that needed 23.67 GiB came down to 13.61 GiB and 11.1 seconds, inside a 13.96 GiB and 11.5 second budget. For H100 serving, the goal was 7,330 tokens generated each second on every GPU, up 50 percent from the parent’s 4,922.

Against the authors’ own adapted Minitron baseline, average accuracy across eight benchmarks rose 6.57 points on the RTX 5070, 2.35 on the RTX 5080, and 0.57 on the H100. The search also used two to 3.3 times fewer tokens. The gain is smallest on the H100, and the RTX 5070 scores are low in absolute terms: AutoCompress averaged 26.67 against the baseline’s 20.10, and scored 0.42 on the AIME25 math exam. A model squeezed into 4 GiB wins its comparison and still answers few hard questions.

Nemotron 3.5 Super needed roughly 280 GB to serve at that context length, so it could not fit on a B200 at all. The pruned model averaged 54.74 across nine benchmarks against the parent’s 57.58, a drop of 2.85 points, which the authors describe as about 95 percent of the parent’s score. It also beat the 120B Nemotron 3 Super reference model, which averaged 45.44. Search and recovery took 128 GPUs about 64 hours and 120 billion training tokens, and the team skipped the Minitron comparison at this size because of that cost. That makes the 74B model a single data point with no like-for-like comparison.

Every model in the study is an NVIDIA model, run on NVIDIA hardware, and the benchmark numbers are the authors’ own. The preprint does not report independent replication.

A 120B model that needed about 280 GB now runs in 178 GB, so any team pricing a deployment around one B200 should test the 74B model on its own tasks before trusting a 2.85-point average.

Reported by arXiv (paper 2611.08290, NVIDIA researchers) on 5 November 2026.