This post summarizes LightMem: Lightweight and Efficient Memory-Augmented Generation.
The easiest way to give an LLM agent memory of past conversation is to stuff the whole history back into the prompt every time. That approach falls apart as conversations grow. A long context triggers the “Lost in the Middle” problem, where information buried in the middle gets ignored, and memory systems that re-read the accumulated history on every turn pay for it with higher compute and slower responses. LightMem targets both problems at once. It’s a lightweight memory-generation system that cuts token usage to a fraction of what existing systems need, while outperforming them.
A Three-Tier Structure Modeled on Human Memory
LightMem’s architecture draws on the Atkinson-Shiffrin memory model from 1968. Just as human memory splits into sensory memory, short-term memory, and long-term memory, LightMem processes conversation history through three matching layers: sensory memory, topic-aware short-term memory, and long-term memory. Each layer plays a different role: filtering, grouping, and refining.
Sensory Memory: Stripping Out Dead Weight First
The first stage is pre-compression. The goal is simple: strip out unimportant information from a document or conversation as early as possible, cutting down what needs to flow through the rest of the pipeline.
Compression starts by running the input through LLMLingua-2, token by token. Each token gets a retain probability, a score for how worth keeping it is, and any token scoring below a threshold ($\tau$) gets dropped without exception. The criterion is information content: tokens with high entropy, the hard-to-predict, information-dense ones, get kept first.
This filtering step removes 50 to 70 percent of all tokens, and semantic coherence survives intact. What matters isn’t token count. It’s how much information those tokens actually carry.
Topic Segmentation: Splitting Conversation by Meaning
Once compressed sensory memory gets batched by turn count, topic segmentation kicks in: detecting where a conversation or document shifts subject and splitting the data there. LightMem combines two signals for this, attention and embedding similarity.
The first signal is the attention-based boundary candidate ($B_1$). The LLM computes an attention score for each conversational turn, and any turn where that score hits a local maximum gets flagged as a topic shift.
The second signal is the embedding-similarity boundary candidate ($B_2$). When the embedding similarity between adjacent turns drops below a threshold ($\tau$), that point also gets flagged as a shift.
The final topic boundary ($B$) is the intersection of both candidates.
$$ B = B_1 \cap B_2 $$
Each signal covers what the other might miss on its own. Unlike splitting at fixed intervals, boundaries decided this way capture the specific context each document or conversation actually has.
Topic-Aware Short-Term Memory
Topic segmentation produces memory fragments split by topic. These get pushed into an STM buffer as $\{topic, \{user_i, model_i\}\}$ pairs, where $user_i$ is the user’s input and $model_i$ is the model’s response.
Once the buffer hits its size threshold, an LLM call summarizes its contents. That summary gets stored alongside the original exchanges in long-term memory as $\{topic, \{sum_i, user_i, model_i\}\}$.
Long-Term Memory: Add Immediately, Clean Up Later
Long-term memory runs on two different rhythms.
Soft update happens during real-time interaction with the user (test-time). New information gets appended immediately, without overwriting or deleting anything already there. That temporarily tolerates duplicates, but it avoids response latency entirely. Compare that to the replace approach most existing systems use, which forces extra computation on every update and pays for it in delay.
Sleep-time offline update runs during idle time, when the model isn’t actively serving inference. This is when LightMem performs parallel consolidation across everything stored in long-term memory: sorting all memories chronologically, then using similarity search to find and merge entries that are semantically redundant or contradictory. An existing entry only gets overwritten if the new one carries a more recent timestamp, which keeps stale information from clobbering something newer.
Appending immediately to kill latency, then batching cleanup into idle time: that combination is what lets LightMem hold onto both response speed and memory quality at once.
Experiments
LightMem was evaluated under Incremental Dialogue Turn Feeding, an environment that mimics how turns actually arrive in a live conversation, one at a time. The pre-compressor was LLMLingua-2, running on a lightweight BERT architecture, with the sensory memory buffer set to 512. Evaluation ran on LongMemEval and LoCoMo, against Full Text, Naive RAG, LangMem, A-MEM, MemoryOS, and Mem0 as baselines.
Under identical model conditions, LightMem outperformed every comparison system while using a fraction, in some cases just a few percent, of their token budget. Latency came out dramatically ahead as well.
How the Retention Ratio Affects Performance
The experiments also varied the retention ratio ($r$), which controls how many tokens survive pre-compression. A higher $r$ lowers the threshold ($\tau$), letting more tokens through. Across the board, $r = 0.6$ generally performed best, though the ideal value shifted with STM buffer size: smaller buffers favored 0.6, while larger buffers did better at 0.7.