Together AI, the inference provider that hosts open-source models for customers, says a new system called Together Compiler can retune how a model is served in under 24 hours, and that the result runs up to 2.6 times faster. The company describes the work in a post on its own blog by Reyna Abhyankar, Dylan Lim, and Simran Arora. The pitch is that serving AI is now too varied for engineers to tune by hand.
Together AI lays out the variation in three layers. None of the ten most popular open-source language models on OpenRouter share an architecture. Across its customer base, Together pushes trillions of tokens daily, and every account sends traffic shaped differently. And the chips range from H100s and B200s to Groq hardware, each needing its own approach.
Traffic also drifts. Together says prompts during one country’s working hours ran 1.4 times longer than another’s, 97,000 tokens against 70,000. After a new model release, prompt length rose 80 percent within two weeks as users moved from chatting to agent-style work. Bursts can double prompt length and drop cache hit rates.
The obvious fix is to let AI agents search for optimizations. The snag, per the post, is experiment speed. Most agent setups restart an inference engine such as vLLM, SGLang, or TRT-LLM for every test, and Together’s table puts a cold start at 467 seconds. Together Compiler keeps one “persistent engine” running and hot-swaps each candidate into it, which Together measures at 9 seconds to be serving. Several agents feed ideas to those engines and trade notes on a shared message board.
A second design choice deals with test accuracy. Optimizing a kernel (a small GPU routine) alone is fast, but the fake inputs often used there can pick the wrong winner for models that route work between experts. Together runs two tiers: fast engines with a few layers for quick, steady timings, and slower engines with many layers to confirm a change holds on realistic data. A cost model then picks how the work is split across GPUs, combining hardware specs, microbenchmarks, and measured runs.
The numbers come from Together’s own testing. It reports up to 2.6 times higher tokens per second across four open models (Kimi K3, DeepSeek v4.1-Flash, GLM 5.3, and MiniMax M3.1-Flash), starting from the best published open-source configurations. At batch size 1, it reports 1205 tokens per second with DeepSeek v4.1-Flash, plus 1111 with the MiniMax model. It calls both the best results yet on the low-latency track of SemiAnalysis’ InferenceX benchmark. On the Llama 1B megakernel test, it says it beats open-source megakernels by 1.7 times. It credits a 252 times higher experiment rate than baseline agent systems.
The post also compares against Claude Code running its /goal command on open-source projects in weather modeling, PDE solving, time series, and biology. Together says its system found bigger speedups on three of four. No independent party has reproduced any of these figures, and the post does not say how many GPU hours each 24-hour run consumed.
Together says the compiler already serves trillions of tokens in production. That matters for anyone buying hosted open models, because it implies that an inference provider’s speed is becoming a continuously updated process rather than a one-time tuning job. Teams comparing hosts on price per token should ask how often each provider re-tunes for their actual traffic, since a gap of 2.6 times on the same hardware would swamp most discount negotiations.
Reported by the Together AI blog; the post carries no publication date in the source.
