The Z.ai research team has published a paper describing Single-rollout Asynchronous Optimization, a reinforcement learning method built to stabilize how large language models are post-trained after they leave pretraining. The method targets a specific weakness in GRPO, the group-based sampling scheme most labs currently use to post-train reasoning and agentic models. Z.ai’s paper argues that scheme was never designed for the asynchronous training loops that long-horizon agent tasks now demand.
GRPO works by sampling several rollouts, or attempted completions, for the same prompt and comparing them against each other to compute a reward baseline. That grouping only makes sense if every rollout in the batch finishes around the same time, which fits a synchronous training loop where the system waits for a full batch before updating weights. Agentic tasks break that assumption. A model attempting a multi-step coding or reasoning task can take wildly different amounts of time to finish depending on how many tool calls or reasoning steps it needs, so forcing a group to complete together stalls the pipeline on its slowest member.
Asynchronous RL was supposed to fix that by updating the model continuously as individual rollouts arrive rather than waiting for a batch. According to the paper, most systems built this way have optimized for throughput while leaving training stability and task effectiveness underexplored. Group-wise sampling, the paper states, simply does not fit that asynchronous setup.
Single-rollout Asynchronous Optimization, which the researchers call SAO, removes the group entirely and samples one rollout per prompt instead. That change reduces what the paper calls off-policy effects, the growing mismatch between the policy that generated a rollout and the policy currently being updated, a gap that widens the longer a system waits to assemble a full group asynchronously. Because there is no group left to compute a relative baseline from, the team pairs single-rollout sampling with a trained value model to estimate that baseline directly. To keep updates from destabilizing training, they also add a strict double-sided, token-level clipping strategy that constrains how far any single token’s update can move the policy in either direction.
The paper reports that SAO trained stably for one thousand steps and consistently beat GRPO and its variants on agentic coding and reasoning benchmarks, including SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. The team also tested the method in a simulated online learning setting where the model has to keep adapting as its environment changes, a scenario closer to production deployment than a fixed benchmark run. SAO was used to train the open GLM-5.2 model, a mixture-of-experts system with 750 billion total parameters and 40 billion active per forward pass, according to the paper.
The benchmark comparisons come entirely from Z.ai’s own paper, run against its own training pipeline, not from an independent evaluation. The paper does not disclose wall-clock or compute-cost comparisons against GRPO, so it is not yet possible to say whether single-rollout training is cheaper in practice or simply more stable at the same cost.
For teams building agentic RL post-training pipelines on open-weight models, the practical bottleneck this addresses is real: waiting on the slowest rollout in a group is an infrastructure tax that scales badly once tasks involve variable-length tool use. If SAO’s stability claims hold up under independent replication, teams currently running GRPO-based async training should treat single-rollout sampling with value-model baselines as a candidate default for their next agentic RL run, rather than a niche optimization specific to Z.ai’s pipeline.
Based on “Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning,” a preprint by the Z.ai research team posted to arXiv on July 8, 2026.
