Decagon has put software agents in charge of making its own AI models run faster, and it has published a table of what they found. In a post on X dated 28 October 2026, authors Pavi Dhiman and Cyrus Asgari report throughput gains per GPU ranging from 1.05x to 2.50x across the company’s production endpoints. Decagon serves more than a dozen of them.
The system is called AutoResearch. Pointed at an endpoint, it measures the current deployment, hunts for the bottleneck, proposes a change, and tests it on real GPUs. If the change wins, the winning setup becomes the new starting point and the cycle repeats. The search covers serving settings, how models are laid out across GPUs, speculative decoding (a trick where a small drafter model guesses upcoming words and the main model checks them), and edits to the code of the inference engine itself. Devin Cloud hosts the loop, and Modal supplies the GPUs.
The part most teams could borrow is the test bench. Decagon replays real production requests, with personal data redacted, at rising arrival rates. Its Rust runner keeps sending on schedule even when the server falls behind, so queues form the way they would in production. The authors also compare token lengths, cache hits, and request timing against live traffic, because matching the average request rate alone can hide a different bottleneck. Before testing anything, the team replays the untouched baseline several times to measure ordinary run-to-run wobble. A candidate has to beat that wobble and repeat the win.
On top of the bench sits an orchestrator agent that spawns one subagent per hypothesis. Each subagent returns an explanation, the exact change, a prediction, and the result that would prove the explanation wrong. Only the orchestrator runs experiments, and every run ends in one of four verdicts: better, same, worse, or invalid.
The headline numbers come from different starting points. The best result, 2.50x, paired FP8 weight quantization with NGRAM speculative decoding. A bundle of engine patches reached 1.99x, and prefill-decode interval tuning reached 1.65x. The smallest gain, 1.05x, came from rejection sampling on an existing draft head, with no training. On one Gemma 4 26B-A4B endpoint that Decagon had already fine-tuned, adapting a DFlash2 drafter lifted throughput by 34 percent over the production drafter. Two other figures in the post use a different unit. Acceptance length, the number of drafted words the main model keeps per check, was about 17 percent higher for the off-the-shelf DFlash2 checkpoint than for the production drafter, and further training raised it another 14 percent over that starting checkpoint. Those are acceptance-length measurements, separate from the 34 percent throughput gain.
Each row is measured against its own baseline, which was either SGLang’s defaults or Decagon’s current tuned configuration. The table does not name the endpoints or give absolute requests per second. Every figure is Decagon’s own.
Quality is handled by people. The post separates lossless changes, such as scheduling, from lossy ones, namely shrinking the numeric precision of the model’s weights or its working memory, which can degrade answers. A human reviews that trade-off and approves each pull request before anything reaches production. Campaigns also carry caps on GPU-hours and concurrent GPUs, and new training code gets a 50-step rehearsal before a full launch. A shared memory of past experiments, including failures, stops the loop from repeating dead ends.
The loop is still started by a person. The authors say they expect agents to begin it themselves when a model update or traffic shift calls for a fresh look.
The gains run from 1.05x to 2.50x, and Decagon’s table does not say which endpoints sit at which end. That spread is the number to ask about before expecting an agent to cut your own inference bill.
Decagon’s post on X by Pavi Dhiman and Cyrus Asgari, published 28 October 2026.
