Context caching: the optimization that actually returns money

A document retrieval assistant was costing approximately four hundred dollars per month in API charges. Six weeks later it was costing six thousand dollars per month. No code changed. No model was upgraded. The usage grew and nobody had set a maximum context window limit.

Long, stable system prompts are the perfect and ideal use case for response caching capabilities. We took a client's three-thousand-token static system prompt and moved it to the cache where it could be reused.

The critical trick to making caching work effectively is prompt ordering: place all stable content first, then variable content last. The cache only works correctly if the stable content never changes on any individual request.

Rearranging templates and reordering content accounted for nearly all the work required. No model change, no architecture modifications, no infrastructure changes whatsoever. The bill dropped to fifty-five percent of its peak within two days.

Worth monitoring the cache hit rate in production carefully because it collapses to zero the moment someone adds a timestamp or any per-request variation into the stable content block.

The same optimization strategy works everywhere stable context exists: system prompts, reference documents, long background instructions, boilerplate preambles that appear in every request.