NVIDIA researchers report that a cheaper way to train small models from big ones often beats the method that has become the default. The paper, from Siyan Zhao, Pavlo Molchanov and nine co-authors, is listed on NVIDIA’s research site.
The setup is distillation: a large “teacher” model coaches a smaller “student.” The popular version, on-policy distillation, has the student produce fresh answers as it improves, with the teacher grading each new batch. The authors test Semi-OPD, where the untrained student writes its answers once, and training reuses that fixed set.
Across 17 teacher-student pairings, spanning 1.5 billion to 235 billion parameters, Semi-OPD won 14. The authors cite gains of up to 13.6% and training up to 11.4 times faster. Those are the paper’s own results; the listing shows no outside replication.
The deciding factor is how closely the two models already agree, measured by overlap in the words they output. Fresh answers pay off only for closely matched pairs. For mismatched ones, the student’s long answers drift away from what the teacher would have written, weakening the feedback.
Teams distilling a small model from a far larger one can measure that overlap first, before paying for the live loop.
Based on a research paper by Siyan Zhao and colleagues, listed on NVIDIA’s research site.
