The Demon in the Cache: KV Cache Design for AI Systems

A research reference on the KV cache for engineers who build inference systems and agent platforms. It borrows a framework from physics, tests it against how inference actually works and against the working month of CodeNinja's own engineers, and turns it into design principles for an organization's AI.

Every AI system is an exchange rate between memory and compute.

Most teams only see one side of it, on the invoice.

In an average month, one of our engineers' coding agents reads ten billion tokens and writes fifteen million.

Almost all of that reading is work the system has already done.

Whether it pays again is decided by how the prompt is ordered, where requests are routed and what the cache is allowed to forget.

Inference is not arithmetic to be made faster.

It is memory to be managed.

WHY THIS MATTERS

The cost of forgetting is invisible until the next request arrives. A cache entry evicted after a few minutes of idleness has to be rebuilt from the prompt text, and the whole prefix is climbed again at full price.

One character can erase a cache. A timestamp or request ID near the front of a system prompt changes every token behind it, silently turning every request into a miss.

Output is the only spend that can never be cached. Every output token climbs the full model alone, one after another, which is why verbose agents are expensive agents.

And at organizational scale, the largest lever is shared. A thousand employees reaching agents that open with the same organization prefix can have it computed once and read by everyone, but only if the prefix is byte-identical everywhere.

This creates a structural exposure.

• Prompts are ordered by habit, so stable material sits behind volatile material and is recomputed on every call.
• Each team writes its own organization preamble, so no two agents share a prefix.
• Load balancers route requests round-robin and throw away cache hits already earned.
• Context windows are filled because they are large, not because the task needs it.
• Eviction is left to recency when it should follow business value.
• The cache is treated as memory, and what the organization learns is lost with it.

Remember deliberately. Forget on purpose.

The cheapest token is the one the system never has to compute twice.

WHAT YOU'LL LEARN

Inside the reference, you'll find:

• Why a thought experiment from nineteenth-century physics is a useful way to reason about AI system design, explained for readers who have never met it
• The primer every new AI engineer needs: tokens, layers, the forward pass, attention, queries, keys and values, and HBM
• Three testable statements about memory and forgetting, and how each holds up against the KV cache
• Where the comparison with physics stops, and why that limit is what makes the framework useful
• Why attention is quadratic, explained with a stamp desk and a dinner party
• The grid: a whiteboard model of the forward pass as vertical work and cache reads as horizontal reads
• What the KV cache removes, what it does not, and what exactly it makes linear
• Why prefill and decode have different bottlenecks, and why they are increasingly served on separate hardware
• How to read the average month of a CodeNinja forward deployed engineer, ten billion input tokens against fifteen million output tokens, as a diagnostic
• How cache hits actually happen, why active sessions stay warm, and why cold starts are drowned rather than avoided
• Ten principles for shaping prompts, infrastructure and agent behavior so the cache works by design
• How to layer a prompt from organization prefix to agent prefix to user context to the current turn
• Which six metrics show the exchange rate between memory and compute directly
• The six anti-patterns that quietly turn a working cache into an expensive one
• Why the cache is built to forget, and where an organization's memory has to live instead

DOWNLOAD THE RESEARCH REFERENCE

Read the complete reference on the KV cache, from the physics of forgetting to the grid, the economics of prefill and decode, a real invoice read as a diagnostic, and a ten-principle playbook for building an organization's AI so the cache works by design.

Every AI system decides what to remember, what to forget and what to recompute. The organizations that make those decisions on purpose will spend less on every turn, and keep more of what each turn teaches them.