One Reordered Tool Can Blow Up Your Coding Agent's Bill

A coding agent resends nearly its entire conversation on every single turn: system prompt, tool definitions, every prior tool call and result, plus a small sliver of new material. Earendil, the team behind the Pi coding agent, published an engineering breakdown last week explaining why that repetition is usually cheap and exactly how it stops being cheap. The mechanism responsible is prompt caching, and the post catalogs eight concrete ways teams accidentally defeat it.

The underlying object is the KV cache, the key and value data a transformer builds for each token while reading a prompt, kept around so later tokens can reference them instead of recomputing from scratch. When a fresh request opens with tokens identical to an earlier one, the inference system reuses that stored work and only has to process whatever comes after the match. Wording that reads the same to a person can still tokenize into a different sequence of numbers, and a single edited token partway through a prompt makes the cached state useless from that point forward.

Providers implement reuse two ways, according to Earendil. Session affinity pins a cache to whichever GPU produced it and routes the next request back to that same worker, which is fast but fragile: the worker can restart, get overloaded, or simply lose priority to a load balancer optimizing for fleet balance over one session’s continuity. Distributed caching instead stores KV blocks in a separate memory tier so a request is not locked to one machine, trading that fragility for a harder indexing and eviction problem. Anthropic’s API layers on explicit cache_control breakpoints that a client must place manually, while other providers locate a reusable prefix automatically with no client-side markers at all.

Time limits are the first failure mode worth checking today. Anthropic’s default cache retention runs five minutes, shorter than a typical test run, build, or lunch break. Earendil notes that Claude Code’s own codebase extends that window to one hour for subscription users, a difference API users can only get by opting into longer retention and paying extra for it. Step away from a session for ten minutes and the next message, however trivial, gets billed as though no cache existed at all.

Tool definitions cause the second failure mode. Names, descriptions, and schemas for every available tool get baked into the system prompt, so adding one tool, dropping one, or simply listing them in a new order shifts the earliest point of mismatch toward the very front of the request. Everything downstream of that shift, potentially tens of thousands of tokens of prior conversation, gets reprocessed at full price. This hits MCP-style tool catalogs and plugin systems particularly hard when they load tools only once they become relevant, since the available loadout keeps changing mid-session. A handful of newer model APIs now let an agent register a tool partway through a conversation without disturbing anything sent earlier, a pattern Earendil labels additive loading, but models lacking that support fall back to resending the full tool list every time, which functions correctly while still wiping the cache.

Branch navigation is the third. Agents that let a user rewind or fork a conversation, as Pi’s tree-structured sessions do, can carry one session ID across what are really several separate prompt histories that share only a fraction of their prefix. Jumping back to an early point in that tree might retain nothing beyond the opening system message and whichever tools were active at the very start, even though the real branch sits much further along.

Earendil published one real session’s numbers as an illustration: 7.13 million input tokens, a 95 percent cache-hit rate, and a total cost of $6.054, against $0.728 re-billed for 161,744 tokens across two cache misses. The gap between that hit rate and a session full of misses is the entire economic argument for treating cache behavior as a design constraint rather than an implementation detail.

Teams shipping their own coding agents or MCP tool servers should audit whether their tool list changes shape mid-session, whether idle gaps regularly exceed five minutes, and whether rewind or branch features silently reset the prefix before assuming a slow, expensive turn is a model problem.

Earendil published this analysis of prompt caching in coding agents on July 22, 2026.