Caching and Prompt Reuse Are the Cheapest AI Cost Cuts You Have

The first reaction most teams have to a surprising AI bill is to go shop for a cheaper model. That's usually the wrong first move. Before you touch the model, look at how many tokens you're paying to re-send that never change between calls. In our experience running production AI features, that's where the real waste lives, and it's fixable without retraining anyone or re-architecting anything.

Take a support-ticket classifier with a 2,000-token system prompt describing categories, tone rules, and a handful of few-shot examples. That prompt gets sent fresh on every single ticket, even though 95% of it is identical call to call. At a few dollars per million input tokens, one call is a rounding error. At 50,000 tickets a day, it's a recurring line item that scales with volume and never gets cheaper just because you picked a smaller model.

The fix isn't one technique. It's three layers of caching that catch different kinds of repetition, and most teams have implemented zero of them because nobody flagged it as the cost problem it is.

Where the tokens are actually going

Almost every production prompt has a large stable part and a small variable part. The stable part is your system instructions, your tool schemas, your few-shot examples, and often a chunk of retrieved context that doesn't change between requests in the same session. The variable part is the user's actual message. Paying full input-token price to re-send the stable part on every call is the waste, and it's invisible until you break down a bill by prompt section instead of by call.

Exact-match caching catches the boring repeats

If the same input produces the same output every time, hash the input and cache the response. This is the simplest layer and the easiest to skip building because it feels too obvious to matter. It matters most for idempotent operations: reprocessing the same document twice because a retry fired, classifying an input that's already been classified, or re-embedding a chunk of text that hasn't changed since last time. In an OCR pipeline feeding a vector store of embedded chunks, re-embedding unchanged chunks on every reindex run is pure cost with zero benefit — a content hash against the last-embedded version skips the call entirely.

function cacheKey(input, tenantId) {
  return `${tenantId}:${sha256(normalize(input))}`;
}

// on request:
const key = cacheKey(prompt, tenantId);
const cached = await store.get(key);
if (cached && !isExpired(cached)) return cached.response;
const response = await callModel(prompt);
await store.set(key, response, { ttl: ttlFor(inputType) });

The normalization step matters more than it looks. Timestamps, request IDs, or whitespace differences in an otherwise identical prompt will break every cache hit if you hash the raw string. Strip anything that doesn't affect the answer before you hash.

Prefix caching turns your system prompt into a discount

Most current model APIs support caching a prefix of the prompt when it's byte-identical across calls. The practical implication is about ordering: put everything stable — system instructions, tool definitions, retrieved context that doesn't change per turn — at the start of the prompt, and put the variable user turn last. If you interleave a session ID, a timestamp, or a random ordering of retrieved chunks near the top, you invalidate the cache on every single call and pay full price without knowing it.

This is also a strong argument for prompt discipline generally. We've written before about the tradeoffs between fine-tuning, retrieval, and just writing a better prompt, and prefix caching is one more reason a stable, well-structured prompt beats a clever one assembled fresh each time. A prompt that changes shape between calls can't be cached no matter how good the wording is.

Semantic caching handles the paraphrase problem

Exact-match and prefix caching both miss the case where two users ask the same question in different words — "what's your return policy" versus "how do I return an item." A semantic cache embeds the incoming query, runs a similarity search against past query-answer pairs, and returns the cached answer if similarity clears a threshold, typically somewhere around 0.90–0.95 cosine similarity depending on how much risk you can tolerate.

This works well for FAQ-style and retrieval-backed answers where the underlying facts change slowly. It's the wrong tool for anything where a small wording difference changes the correct answer — "can I return this after 30 days" and "can I return this after 60 days" will embed close together and a naive threshold will happily return the wrong policy. If you build this, tune the threshold against real query pairs, not a guess, and treat cache hits as a category worth including in whatever evaluation process you're already running. We've covered building an eval loop without a labeled dataset in evaluating an AI feature with no labelled dataset, and the same approach applies to catching bad cache hits before customers do.

The honest tradeoff: staleness and tenant isolation

Caching trades freshness for cost. If the underlying data changes and the cache doesn't know, you serve a confidently wrong answer. Set TTLs to match how fast the source data actually moves, not a round number that felt safe. A cached answer about a return policy that changes twice a year can live for days. A cached answer about current stock on hand in a point-of-sale system should not be cached at all, or should live for seconds — we learned this running offline-capable inventory sync where staleness has a direct dollar cost the moment a customer buys something that's no longer in the store.

The tenant isolation issue is less obvious and more dangerous. If your product serves multiple tenants — and across our own products and client platforms we've operated more than 700 tenant databases — a cache key that doesn't include the tenant ID will eventually leak one tenant's cached answer to another. This isn't a hypothetical edge case; it's the default failure mode of any cache built without multi-tenancy in mind from the start. Every cache key needs the tenant ID baked in, and any cache entry derived from tenant-specific retrieved context needs to be scoped to that tenant, full stop.

A decision rule for what to cache

• Deterministic input, high call volume, identical output expected → exact-match cache with content-hash keys.
• Large stable prompt section reused across many calls (system prompt, tool schemas, retrieved context) → prefix caching, with stable content ordered first.
• Free-text queries with overlapping intent and slow-changing answers → semantic cache with a tuned similarity threshold, reviewed periodically.
• Answer depends on data that changes faster than your cache TTL → don't cache, or cache with a TTL tied to that data's actual volatility.
• Multi-tenant system → tenant ID is part of every cache key, no exceptions.
• Answer depends on per-user permissions → cache key includes the permission context, or don't cache it.

Where the savings actually land

Caching doesn't just lower the bill, it changes what your budget controls can afford to be strict about. If you're running AI spend wallets with per-tenant budgets and auto-recharge, a lower baseline burn rate means you can set tighter per-tenant caps without throttling legitimate usage, because the calls that would have hit the ceiling are the ones now getting served from cache instead of the model. That's a better lever than raising every tenant's limit and hoping usage stays reasonable.

The cheapest token is the one you never send twice.

If you're scoping this kind of work, our approach covers how we sequence cost controls against everything else in an AI feature build, and the portfolio has the production systems this thinking came out of. Before you spend a sprint evaluating a cheaper model, spend an afternoon measuring your cache hit rate. It's very often the bigger number.