AI agent memory explained: short-term, long-term and semantic memory

A language model remembers nothing between requests. Everything an AI agent appears to "remember" is text your application chose to send again: recent messages, a summary of older ones, facts about the user, or notes found by searching a store. Designing agent memory is therefore a data design job: what to keep, where, for how long, how to find it again, and how to delete it when a user asks. This guide explains short-term, long-term and semantic memory, what the major providers now offer, and includes a tested memory layer in TypeScript with a working deletion path.

Tested with: Node.js 24.19.0 (built-in node:sqlite, TypeScript run directly), TypeScript 6.0.3 (tsc --noEmit, strict) and @anthropic-ai/sdk 0.116.0. The storage, recall and deletion code ran locally; the Claude summariser was type-checked but not run against a live key.

Three kinds of memory

The terms come from psychology and are used loosely, so here is how they map to software:

Researchers sometimes add episodic memory (records of past tasks and their outcomes) and procedural memory (how to do things, usually the system prompt and tools). In most business systems episodic memory is a conversation log plus summaries, and procedural memory is your prompt and code, so the three types above cover the design decisions.

Short-term memory: the context window

The context window is the only memory the model actually reads. Windows are now large: Claude Sonnet 4.6 accepts a 1M-token context at standard prices, according to Anthropic's pricing page. That does not mean you should fill it. Every token is billed on every request, latency grows with input length, and models use information in the middle of long inputs less reliably than at the start or end, a pattern documented in Lost in the Middle (Liu et al.).

So treat the window as a budget. The common pattern is:

1. Keep the most recent turns verbatim, as many as fit the budget.
2. Fold older turns into a rolling summary that keeps decisions, open questions, identifiers and stated preferences.
3. Put durable facts and recalled notes in their own clearly labelled sections, so the model can tell memory from the current request.

Providers now offer server-side help with this:

• Anthropic: compaction (beta) summarises earlier context automatically when a conversation approaches a trigger threshold, and context editing (beta) clears old tool results or thinking blocks. You must send the returned content back unchanged so the compaction state is kept.
• OpenAI: the Responses API can chain turns with previous_response_id or a durable Conversations object. Per the conversation state guide, response objects are stored for 30 days by default (set store to false to opt out), conversation items have no 30-day limit, and earlier tokens in a chain are still billed as input.

Server-side state is convenient, but it moves your users' data into another system with its own retention rules. Know where it lives before you rely on it.

Long-term memory: a small set of explicit facts

Long-term memory works best when it is boring: a short list of named facts with an owner, a source and a timestamp. Two rules prevent most problems:

• Allowlist what can be remembered. "Preferred name", "timezone" and "communication preference" are useful. Health details, payment data and opinions about the customer are not, and storing them creates legal and security risk.
• Let users see and edit it. If the agent remembers something wrong, the user should be able to correct it. A table of keys and values makes that easy; a pile of free text does not.

Anthropic's memory tool takes a file-based approach: you declare memory_20250818, Claude issues view, create, str_replace, insert, delete and rename commands against a /memories directory, and your application executes them against storage you control. It is available on Claude 4 and later models, and the whole configuration is one entry in tools: {"type": "memory_20250818", "name": "memory"}.

Because the tool runs client-side, the documentation is explicit that safeguards are yours: validate every path to stop traversal such as /memories/../../secrets.env, cap file sizes, expire old files and keep sensitive data out. For agents hosted on Anthropic's platform, Managed Agents now also offers memory stores (public beta), which are versioned so that a leaked value can be deleted and its earlier versions redacted.

Semantic memory: recall by meaning

Semantic memory stores free-text notes with an embedding, a vector that represents meaning, and retrieves the closest notes for each new message. This is the same machinery as retrieval-augmented generation, applied to what you learned about a user rather than to your documents; our RAG pipeline tutorial covers embeddings and reranking in depth.

Three settings decide whether it helps or hurts:

• A minimum similarity score. Returning the top three notes even when none is relevant fills the context with noise the model may act on.
• An expiry date. A note that a customer was moving house is useful for a month, not for five years.
• Per-user scoping in the query itself. Filter by user ID inside the database query, never after retrieval, or one customer's notes will eventually appear in another customer's answer.

A working memory layer

The class below stores all three kinds of memory in SQLite using Node's built-in node:sqlite module, so it runs with no database server. It builds the context for each request within a token budget, refuses facts that are not on an allowlist or look like card numbers or keys, recalls notes above a similarity threshold with an expiry date, and deletes everything for a user in one transaction.

import { DatabaseSync } from 'node:sqlite';

export type Turn = { role: 'user' | 'assistant'; content: string };
export type Embed = (text: string) => Promise<Float32Array>;

// Rough budget check; use the provider's token-counting endpoint when you need exact numbers.
const approxTokens = (s: string) => Math.ceil(s.length / 4);
const SECRET = /\b(?:\d[ -]?){13,19}\b|sk-[A-Za-z0-9_-]{20,}|-----BEGIN [A-Z ]*PRIVATE KEY-----/;
const ALLOWED_FACTS = new Set(['preferred_name', 'language', 'timezone', 'plan', 'communication_preference']);

export class MemoryStore {
  private db: DatabaseSync;
  private embed: Embed;
  constructor(path: string, embed: Embed) {
    this.embed = embed;
    this.db = new DatabaseSync(path);
    this.db.exec(`
      CREATE TABLE IF NOT EXISTS turns   (user_id TEXT, conv_id TEXT, seq INTEGER, role TEXT, content TEXT, PRIMARY KEY (conv_id, seq));
      CREATE TABLE IF NOT EXISTS summaries (conv_id TEXT PRIMARY KEY, user_id TEXT, summary TEXT, upto_seq INTEGER);
      CREATE TABLE IF NOT EXISTS facts   (user_id TEXT, key TEXT, value TEXT, source_conv TEXT, updated_at TEXT, PRIMARY KEY (user_id, key));
      CREATE TABLE IF NOT EXISTS notes   (id INTEGER PRIMARY KEY, user_id TEXT, text TEXT, vector BLOB, created_at TEXT, expires_at TEXT);`);
  }

  // Short-term memory: the conversation log.
  addTurn(userId: string, convId: string, turn: Turn) {
    const row = this.db.prepare('SELECT COALESCE(MAX(seq), -1) + 1 AS next FROM turns WHERE conv_id = ?').get(convId) as { next: number };
    this.db.prepare('INSERT INTO turns VALUES (?, ?, ?, ?, ?)').run(userId, convId, row.next, turn.role, turn.content);
  }

  // Build the context for the next request: summary of old turns + as many recent turns as fit the budget.
  async contextFor(convId: string, budgetTokens: number, summarise: (old: Turn[], previous: string) => Promise<string>) {
    const all = this.db.prepare('SELECT seq, role, content FROM turns WHERE conv_id = ? ORDER BY seq').all(convId) as unknown as (Turn & { seq: number })[];
    const saved = this.db.prepare('SELECT summary, upto_seq FROM summaries WHERE conv_id = ?').get(convId) as { summary: string; upto_seq: number } | undefined;
    let summary = saved?.summary ?? '';
    let upto = saved?.upto_seq ?? -1;
    const recent: Turn[] = [];
    let used = approxTokens(summary);
    for (let i = all.length - 1; i > upto; i--) {
      const t = approxTokens(all[i].content);
      if (used + t > budgetTokens) break;
      recent.unshift({ role: all[i].role, content: all[i].content });
      used += t;
    }
    const firstKept = all.length - recent.length;
    const overflow = all.filter((t) => t.seq > upto && t.seq < firstKept);
    if (overflow.length) {
      summary = await summarise(overflow, summary);
      upto = overflow[overflow.length - 1].seq;
      this.db.prepare('INSERT INTO summaries VALUES (?, (SELECT user_id FROM turns WHERE conv_id = ? LIMIT 1), ?, ?) ON CONFLICT(conv_id) DO UPDATE SET summary = excluded.summary, upto_seq = excluded.upto_seq')
        .run(convId, convId, summary, upto);
    }
    return { summary, recent, summarisedTurns: upto + 1 };
  }

  // Long-term memory: a few explicit, allowlisted facts the user can see and edit.
  remember(userId: string, key: string, value: string, sourceConv: string) {
    if (!ALLOWED_FACTS.has(key)) throw new Error(`fact "${key}" is not on the allowlist`);
    if (SECRET.test(value)) throw new Error('refusing to store something that looks like a secret or card number');
    this.db.prepare('INSERT INTO facts VALUES (?, ?, ?, ?, ?) ON CONFLICT(user_id, key) DO UPDATE SET value = excluded.value, source_conv = excluded.source_conv, updated_at = excluded.updated_at')
      .run(userId, key, value, sourceConv, new Date().toISOString());
  }
  facts(userId: string) {
    return this.db.prepare('SELECT key, value FROM facts WHERE user_id = ? ORDER BY key').all(userId) as { key: string; value: string }[];
  }

  // Semantic memory: notes recalled by meaning, with an expiry date.
  async note(userId: string, text: string, ttlDays = 180) {
    if (SECRET.test(text)) throw new Error('refusing to store something that looks like a secret or card number');
    const v = await this.embed(text);
    const now = new Date();
    this.db.prepare('INSERT INTO notes (user_id, text, vector, created_at, expires_at) VALUES (?, ?, ?, ?, ?)')
      .run(userId, text, new Uint8Array(v.buffer), now.toISOString(), new Date(now.getTime() + ttlDays * 864e5).toISOString());
  }
  async recall(userId: string, query: string, k = 3, minScore = 0.3) {
    const q = await this.embed(query);
    const rows = this.db.prepare('SELECT text, vector FROM notes WHERE user_id = ? AND expires_at > ?').all(userId, new Date().toISOString()) as { text: string; vector: Uint8Array }[];
    return rows
      .map((r) => ({ text: r.text, score: cosine(q, new Float32Array(r.vector.buffer, r.vector.byteOffset, r.vector.byteLength / 4)) }))
      .filter((r) => r.score >= minScore)
      .sort((a, b) => b.score - a.score)
      .slice(0, k);
  }

  // Right to erasure: remove everything held about a user, in one transaction.
  forgetUser(userId: string) {
    const tables = ['turns', 'summaries', 'facts', 'notes'] as const;
    const counts: Record<string, number> = {};
    this.db.exec('BEGIN');
    try {
      for (const t of tables) counts[t] = Number(this.db.prepare(`DELETE FROM ${t} WHERE user_id = ?`).run(userId).changes);
      this.db.exec('COMMIT');
    } catch (e) { this.db.exec('ROLLBACK'); throw e; }
    return counts;
  }
}

function cosine(a: Float32Array, b: Float32Array) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return na && nb ? dot / Math.sqrt(na * nb) : 0;
}

The summariser calls Claude to fold older turns into the running summary, and tells it what must survive and what must never be written down:

import Anthropic from '@anthropic-ai/sdk';
import type { Turn } from './memory.ts';

const client = new Anthropic();

// Rolling summary: fold the turns that no longer fit into the previous summary.
export async function summariseWithClaude(old: Turn[], previous: string): Promise<string> {
  const transcript = old.map((t) => `${t.role}: ${t.content}`).join('\n');
  const res = await client.messages.create({
    model: process.env.CLAUDE_MODEL ?? 'claude-sonnet-4-6',
    max_tokens: 600,
    system: 'Update the running summary of a support conversation. Keep decisions, open questions, order numbers and stated preferences. ' +
      'Drop greetings and small talk. Never include payment card numbers, passwords or other secrets. Under 200 words.',
    messages: [{ role: 'user', content: `Previous summary:\n${previous || '(none)'}\n\nNew turns:\n${transcript}` }],
  });
  return res.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('').trim();
}

To test the mechanics without network access, the test script swaps in a hashed bag-of-words embedder and a placeholder summariser. These are clearly test stand-ins; in production you pass a real embedding model and summariseWithClaude.

// Offline test of the memory layer. The embedder and summariser here are deterministic stand-ins so the
// test runs without network access; production uses a real embedding model and summariseWithClaude().
import { MemoryStore, type Turn } from './memory.ts';

const words = (s: string) => s.toLowerCase().match(/[a-z]+/g) ?? [];
const testEmbed = async (s: string) => {            // hashed bag of words, 256 dimensions: test only
  const v = new Float32Array(256);
  for (const w of words(s)) { let h = 0; for (const c of w) h = (h * 31 + c.charCodeAt(0)) >>> 0; v[h % 256] += 1; }
  return v;
};
const testSummarise = async (old: Turn[], prev: string) => `${prev ? prev + ' ' : ''}[${old.length} earlier turns summarised]`;

const mem = new MemoryStore(':memory:', testEmbed);
for (let i = 1; i <= 12; i++) mem.addTurn('u42', 'c1', { role: i % 2 ? 'user' : 'assistant', content: `Message ${i}: ` + 'details about order 1182 and delivery '.repeat(10) });

const ctx = await mem.contextFor('c1', 500, testSummarise);
console.log(`short-term: ${ctx.recent.length} recent turns kept, ${ctx.summarisedTurns} older turns folded into summary`);

mem.remember('u42', 'preferred_name', 'Sam', 'c1');
mem.remember('u42', 'communication_preference', 'email, not phone', 'c1');
for (const [key, value] of [['card', '4111 1111 1111 1111'], ['communication_preference', 'card 4111 1111 1111 1111']]) {
  try { mem.remember('u42', key, value, 'c1'); } catch (e) { console.log(`rejected ${key}: ${(e as Error).message}`); }
}
console.log('long-term facts:', mem.facts('u42').map((f) => `${f.key}=${f.value}`).join('; '));

await mem.note('u42', 'Customer said the courier left parcels at the wrong door twice in March');
await mem.note('u42', 'Customer prefers eco packaging and no paper invoices');
await mem.note('u42', 'Customer asked about bulk discounts for office chairs');
for (const r of await mem.recall('u42', 'problems with courier delivery to the wrong door')) console.log(`recall ${r.score.toFixed(2)}: ${r.text}`);

console.log('forget user u42:', JSON.stringify(mem.forgetUser('u42')));
console.log('after deletion: facts', mem.facts('u42').length, '| notes recalled', (await mem.recall('u42', 'courier')).length);

The run shows each control working: with a 500-token budget, five recent turns stay verbatim and seven older ones are folded into the summary; a card number is rejected both as a new key and when hidden inside an allowed key; the courier complaint is recalled and the unrelated notes fall below the threshold; and after forgetUser nothing about the user remains in any table.

Notice what the agent never sees: the database. The model receives a prepared context and returns text; your code decides what is written back. That separation is what makes memory auditable, because every stored fact has a source conversation and a timestamp you can show to the user or a regulator.

Production needs a little more: exact token counts from the provider's counting endpoint, a real vector index such as pgvector once you have more than a few thousand notes per user, and encryption at rest. Getting the write rules, retention and deletion right across several stores is where most memory projects stall, and it is the kind of work our AI business process automation team handles.

Privacy and deletion

Memory turns a stateless model into a system that holds personal data, so data protection law applies. Under the GDPR, which covers personal data of people in the EU and, through the UK GDPR, the UK, individuals have a right to erasure (Article 17), and you need a lawful basis and a defined retention period for what you keep. Similar rights exist under laws such as California's CCPA. In practice:

• Map every copy. A user's data may sit in the conversation log, summaries, the facts table, vector notes, backups and the provider's stored responses. Deleting the chat alone is not enough.
• Summaries inherit the data they summarise. If you delete the turns, delete or regenerate the summary built from them.
• Vector indexes need care. Deleting a row may leave the vector in an approximate index until it is rebuilt or compacted. Check your vector database's deletion semantics.
• Provider-side state has its own rules. OpenAI's help centre notes that deleting a ChatGPT chat does not delete saved memories created from it, and that deleted memories may be kept in logs for up to 30 days (Memory FAQ). Your own product needs equally clear rules, written down and tested.
• Never store secrets. Anthropic's memory documentation warns that memories are replayed into future contexts, so a key written once is exposed in every later session. Filter before writing, as the example does.

Memory is also an attack surface. Text that reaches memory from an email or web page can carry instructions that the agent will read again in every later session. Treat stored memory as untrusted input, keep it in a labelled section of the prompt, and apply the patterns in our prompt injection guide.

Common mistakes

• Sending the whole history forever. Costs climb with every turn and quality falls. Budget the window and summarise.
• Letting the model decide what to remember with no rules. You end up storing sensitive or wrong details you cannot explain. Allowlist keys and log the source of each fact.
• No expiry. Stale notes are recalled as if they were current.
• Filtering by user after retrieval. Scope every memory query by user ID inside the database.
• Forgetting derived data. Summaries, embeddings and caches all contain the original data.
• Treating memory as truth. Memory says what someone said at some point. For orders, balances and stock, ask the system of record.

When you do not need memory

Many useful agents need no long-term memory at all. A tool that drafts replies to one ticket, classifies an invoice or summarises a document is better stateless: cheaper, easier to reason about and nothing to delete. Add memory only when users notice its absence, such as repeating their preferences or re-explaining an ongoing case, and start with the smallest kind that fixes it, usually a summary plus a few explicit facts.

Summary

Short-term memory is the context window, managed with a budget and a rolling summary. Long-term memory is a small, allowlisted set of facts users can see and change. Semantic memory is searchable notes with a relevance threshold and an expiry. Provider features such as compaction, the memory tool, managed memory stores and stored conversations reduce the code you write, but they do not remove your responsibility for what is stored and how it is deleted.

Planning an assistant that remembers customers without becoming a privacy risk? Cloudsoft Zone builds AI business process automation with defined memory, retention and deletion built in.