What Your Agent Should Remember, Recompute, or Forget

Most agent frameworks ship a memory module and stop there. Nobody tells you what belongs inside it. Teams default to one of two failure modes: persist everything the agent ever sees, or persist nothing and re-derive the world from scratch on every call. Both feel safe. Both are wrong, and they fail for different reasons.

Persist everything and you get a vector store that grows without bound, full of facts that were true last Tuesday and aren't now. The agent confidently repeats a shipping address a customer changed three weeks ago, or a project timeline that got revised twice since. Persist nothing and the agent re-summarizes a 40-message conversation on message 41, re-fetches documents it already read, and burns latency and tokens rediscovering things a human colleague would just remember.

Neither is really a memory problem. It's a data model problem wearing an AI costume. Once you split "memory" into three separate questions — what to persist, what to recompute, and what to forget — the design gets a lot simpler, and a lot cheaper to run.

The two failure modes teams default to

The store-everything camp usually gets there by accident. Every tool call, every retrieved chunk, every intermediate reasoning step gets written to a table or a vector index because someone was worried about losing information. Six months later the index has ten times the chunks it needs, retrieval quality drops because stale and current facts compete for the same query, and nobody can say with confidence which record is the truth. We've written before about how retrieval quality is fundamentally a chunking problem, not a model problem — the same discipline applies to what you let into memory in the first place.

The store-nothing camp is a reaction to that mess, and it trades one cost for another. Recomputing a conversation summary from raw history on every turn is fine at message five and expensive at message fifty. Re-running a document extraction pipeline because the agent didn't keep the structured result the first time means paying for the same OCR or vision-model call twice. We've covered the cost side of this directly in our piece on caching and prompt reuse as the cheapest lever you have — memory design and cache design are the same discipline pointed at different time horizons.

Three questions to ask before you store anything

Before a fact goes into long-term memory, run it through three checks. If it fails any one of them, it doesn't belong there in its current form.

• Does it change? If the fact is immutable (a customer's date of birth, a signed contract term) it's safe to persist indefinitely. If it's mutable (a preferred shipping address, a project's current phase) it needs a source of truth it can be re-checked against, not just a stored copy.
• Is it cheap to recompute from source? If regenerating the fact costs one cheap query against a system of record, don't cache it as agent memory — just fetch it live. If it costs a model call, an OCR pass, or a multi-step tool chain, persisting the result is the right call.
• Does keeping it create risk? Stale facts that get repeated with false confidence are worse than no memory at all. PII sitting in a vector store past its retention need is a liability, not an asset.

That third question is the one teams skip, and it's the one that gets expensive later.

What to persist

Persist facts that are expensive to derive and don't change on their own — they only change when something in the world changes, and that change should be an explicit event, not a silent overwrite. Good candidates: a customer's confirmed preferences in a loyalty system, the structured output of a vision-model extraction, and any fact a human has explicitly corrected or verified. In Retailer OS we persist loyalty and preference data per customer because re-deriving it from raw transaction history on every interaction would be slow and would still miss the manual corrections a store associate made at the counter — a human correction is a stronger signal than anything the model inferred on its own, which is the whole argument behind designing human-in-the-loop review that people actually use.

A minimal schema for a persisted memory record looks like this. The fields that matter are source and supersedes — they let you tell a verified fact from an inferred one, and let a newer fact explicitly retire an older one instead of silently coexisting with it.

interface MemoryRecord {
  entityId: string;
  fact: string;
  source: 'user_confirmed' | 'tool_output' | 'inferred';
  confidence: number;
  createdAt: string;
  ttlDays: number | null;
  supersedes: string | null;
}

ttlDays is not optional decoration. Every record should have an expiry policy decided at write time, not discovered later during a cleanup sprint.

What to recompute

Recompute anything whose value comes from freshness rather than derivation cost. Conversation summaries are the clearest example: don't store "the summary" as a durable fact, store the raw event log and regenerate the summary on demand from the last N events. The summary is a view, not a record. The same logic applies to a construction takeoff: in BuildAEC, a vision-model quantity takeoff carries a confidence score computed fresh against the current plan revision. If a contractor uploads a revised sheet, the old takeoff doesn't get patched — it gets recomputed, because trusting a stale confidence score against a new plan is worse than paying for a new inference pass.

The decision rule here is simple: if the source data can change independently of the derived fact, and a stale derived fact would be actively misleading rather than just slightly out of date, recompute instead of persist. Cache the recomputation aggressively — that's a caching problem, not a memory problem — but don't let the cached value outlive its own justification.

What to forget, and how to enforce it

Forgetting is a feature, not data loss, and it needs the same rigor as persistence. Three categories should have a forced expiry:

• Session scratch state — anything the agent needed to complete one task and nothing else. Purge it when the session ends, not when someone notices the table is huge.
• Superseded facts — once a new confirmed fact replaces an old one, delete the old one rather than leaving both in the index for retrieval to pick between at random.
• Anything past its compliance retention window — raw documents, PII, and tool outputs containing personal data should have a hard deletion job, not a policy document nobody automated.

A concrete rule that works in practice: every memory record gets a TTL at write time based on its category — session memory dies with the session, unverified inferred facts expire in days, human-confirmed facts expire in months unless reconfirmed, and anything touching PII follows whatever your data retention policy already requires. Write the deletion job before you write the feature that creates the data, not after.

The tradeoff nobody likes: memory has a half-life

Persisting a fact makes the agent faster and cheaper on every future call that needs it. It also makes the agent confidently wrong the moment the world moves and the record doesn't. There is no version of agent memory that avoids this tradeoff — you're always choosing between the latency and cost of recomputation and the risk of staleness. The honest answer is that the choice should be made per fact type, explicitly, not once for the whole system.

A stale fact repeated with confidence does more damage than no memory at all.

This advice doesn't apply everywhere. A single-turn tool that answers one stateless question — parse this address, classify this image — doesn't need a memory layer at all, and adding one is pure overhead. Memory design earns its complexity only when the agent operates across multiple turns or sessions against the same entity. If your agent doesn't have that property yet, don't build the memory system in anticipation of it; build it when the second call actually needs the first call's output.

Getting this split right is most of what separates an agent that feels reliable after a hundred interactions from one that feels reliable in the demo and erratic in production. If you're scoping an agent feature and want a second set of eyes on the memory model before you write the schema, our services page covers how we approach this kind of work, and our approach page walks through how we scope and price it.