Training the largest AI models, systems built from many specialist submodels, can hit a memory ceiling well before compute runs out. A paper posted to arXiv on September 13 identifies four separate parts of that training process, each growing at its own rate: the step that routes data to the right specialist, the layer that scores predictions against the full vocabulary, the checkpoints that hold work in progress, and the optimizer that updates parameters. Whichever bottleneck is largest gets exposed once the others are fixed, the paper argues, so all four need bounding together.
The paper, credited to Xuan Phi Nguyen, proposes one fix per bottleneck: capping how many tokens move through the routing step at once, breaking vocabulary scoring into smaller circulating chunks, offloading a single large tensor per checkpoint to CPU memory, and pipelining the optimizer’s update step. Applied together on models ranging from 120 billion to 667 billion parameters, the combined method reached roughly one million tokens of context, up to 32 times further than a standard baseline, and trained up to 10.4 times faster.
According to a paper submitted to arXiv on September 13, 2026.
