Ramp’s engineering team built a routing system that decides, in real time, which model and which pricing tier is likely to finish a request on time and at the lowest cost. The system now cuts spend by more than a quarter across a large slice of Ramp’s traffic, and by 30 percent on its highest-volume streaming workload, according to a post from Ramp engineer Kedar Thakkar published July 20.
The mechanism matters more than the headline number. Ramp’s gateway tracks two signals for every model it can call: an exponentially weighted moving average of that model’s failure rate, counting only genuine provider-side errors rather than requests a caller sent incorrectly, and a running estimate of its latency distribution. The latency estimate is built with Thompson sampling, a technique that balances trying underused options against sticking with known-good ones. Ramp fits a Normal-Inverse-Gamma prior over the mean and variance of log-latency, then updates that posterior from pooled statistics stored in Redis as new requests complete.
Those two signals feed a single formula before every routing decision. The probability that a given model produces a bad outcome equals its failure probability plus the chance it does not fail but still misses the caller’s deadline. Ramp multiplies that risk against each model’s price, ranks the resulting list, and sends the request to the top option, falling back down the list if a call fails. Callers still set a priority order, so a customer-facing chat feature can prefer Claude Opus 4.8 for tone while accepting GPT 5.6 as a fallback, but the gateway decides in real time whether a cheaper tier can still hit the deadline.
The reranker deployment shows the system working as designed. During a stretch of heavy rate limiting on Gemini 3.1 Flash Lite, the router shifted traffic to GPT 5.4 Nano and reverted once Gemini recovered. It also sent that traffic to GPT 5.4 Nano’s Flex tier rather than its Standard tier, because at that moment both tiers were finishing comfortably inside the reranker’s six-second deadline at nearly identical latency, making the half-price option the correct call. Measured across broader traffic, the approach produced a 26.3 percent cost reduction with a 0.09 percentage point improvement in error rate, meaning the savings came with no reliability tradeoff.
That Flex-tier example is the part worth studying. Most teams treat a provider’s cheaper service tier as a fixed discount with a fixed latency penalty, chosen once at build time and rarely revisited. Ramp’s data, which started with a chart from CTO Rahul Vasa (per the post) and a tip from customer Mercana that flex-tier latency degraded only during business hours, shows that assumption breaks down hour to hour. A router that samples live latency data can capture savings a static configuration leaves on the table.
The bigger point for anyone operating a multi-model gateway is architectural. Ramp is not claiming an edge from picking better models; it is claiming an edge from the layer that sits between callers and every provider, deciding which model and tier to hit under live conditions. As the underlying models continue to commoditize and providers keep adding pricing tiers, that routing and integration layer, not any single model choice, becomes the place where cost and reliability are actually won or lost.
Any team running its own LLM gateway should treat this as a build prompt: track per-model failure rates and latency distributions separately, and revisit tier selection continuously rather than at deploy time.
Reported by Ramp’s engineering blog on July 20, 2026.
