ByteByteGo12 min readintermediate
Do LLMs Have the Memory of a Goldfish?
Summary
The article explains that LLMs don’t have persistent personal memory; all “memory” is supplied by the surrounding application via the context window, summaries, or external storage. It outlines the distinction between trained weights, working‑memory (token context), and persistent application memory, shows how to construct API calls to preserve conversation state, and discusses the cost and laten…
- LLMs only retain information that is explicitly included in the prompt (the context window); they cannot remember across calls on their own.
- Applications must manage conversation history, summaries, and external data stores to give the model the illusion of memory.
- Context windows are a finite token budget; as conversations grow you must prune, compress, or replace older content to stay within limits.
- Longer contexts increase both token cost and latency, and can cause “context rot” where the model struggles to surface the right facts.
Understanding the separation between model weights and application‑level memory is essential for building cost‑effective, low‑latency AI agents that can handle multi‑turn interactions without blowing up token usage.
5/10
