Hugging Face has released “The Ultimate MoE Training Handbook,” a very long guide to training mixture-of-experts models, written by Nouamane Tazi, Leandro von Werra and Thomas Wolf and published on 5 November 2026. The authors put the reading time at one to two days.
A mixture-of-experts model (MoE) is a language model split into many smaller specialist blocks, called experts. For each word fragment it processes, a small component called the router picks a few experts to do the work and leaves the rest idle. The model can therefore store far more knowledge than it spends compute on at any single step. The small model the handbook trains in its second chapter has 6.86 billion parameters in total, and only 1.22 billion of them are active for each token.
The reason Hugging Face wrote it is visible on a leaderboard. Artificial Analysis ranks open-weights models, and the handbook counts 18 mixtures of experts among the top 20. Two dense models break the pattern: Nanbeige4.3-33B ranks twelfth, and Qwen3.8 27B ranks thirteenth. Each of five leading labs has its best entry on that list, among them GLM-5.3, DeepSeek V4.1 Flash 1017, MiniMax-M3.1-Flash, MiMo-V2.6-Pro and Ling 3.1 Flash 1019.
The authors say the cheaper running cost is the easy part. Making training faster is where teams struggle. Switching away from a conventional dense model adds three jobs: the router that assigns tokens to experts, a balanced workload so no expert is overwhelmed while others sit unused, and a constant exchange of tokens between GPUs once the experts live on different chips. Every one comes with fresh settings to tune and fresh ways to crash a training run.
The handbook is built around those failure modes. Its introduction includes a map that starts from a symptom, such as a run going wrong, and points the reader to the chapter that explains the likely cause. The chapters pair explanations with runnable code and, mostly, measurements from the authors’ own cluster.
Chapter 2 gets practical. The commands assume a Slurm cluster (a common job scheduler for shared GPU machines) with eight GPUs per node, and the authors ran theirs on AWS machines with H100 chips. Before any training, they have readers measure how fast the GPUs can swap data, because a healthy general-purpose test does not predict how an MoE will behave. In the authors’ own sweep, throughput for that exchange dropped as the job grew: 77.14 GB/s across two nodes, 39.25 across four, and 30.91 across eight. Readers without a cluster can rent an eight-GPU node through Hugging Face Jobs, which bills by the minute.
The authors name their audience. It is people who have already run a conventional model on real GPU hardware, and who are ready for an MoE. Readers who are curious about the design and about frontier labs’ choices can jump to chapter 3, which takes the layer apart. Chapter 4 follows the same choices into serving and reinforcement learning, according to the introduction.
This is a sequel of sorts. The team’s earlier Smol Training Playbook covered a dense model and touched MoE design only briefly. Only the first stretch of the new handbook was available for this report, so the later chapters’ claims are not assessed here.
For a lab or company weighing whether to train its own model, the handbook turns a skill that sat mostly inside a few frontier teams into something a competent infrastructure group can try on one rented node.
Reported by Hugging Face on 5 November 2026.
