The agent's memory got too big to load — and it failed silently

# The agent's memory got too big to load — and it failed silently

Symptom, in the words you'd type at 1am: a session opens in Claude Code and the assistant is worse than it was yesterday. It re-asks a question you answered a week ago. It walks straight into a trap you paid four hours for. Nothing errored. Nothing warned you. It just knows less.

For context, since the specifics matter if you are trying to reproduce any of this: the project is a solo game built in Unreal Engine 5.8 (Blueprint-only) with Blender 5.1, driven through the Claude Code desktop app for Windows running Opus 5. The memory described here is Claude Code's own project memory — a folder of Markdown notes plus an index file — not anything Unreal provides.

That has happened to me twice in the last week, for two completely different reasons, and untangling them is most of the value in this entry.

---

Two stores, and they are not the same thing#

Almost every confused account of this problem — including my own, mid-panic — collapses two separate things into "the memory".

The knowledge store. Roughly 430 hand-written Markdown notes, one topic each, totalling about 3.7 MB. Traps, recipes, decisions, the state of each subsystem. This is distilled, expensive and irreplaceable — a note in there is typically the residue of an hour or a day of being wrong. Sitting on top of it is a single index file that is injected into the opening context of every session. The index is a pointer list: it does not hold the knowledge, it tells the session which note to open.

The transcript store. The raw conversation logs. On this machine that is 1.31 GiB across about 1,900 files — 542 conversation logs plus sidecars. It is a byproduct. It exists so you can resume a session.

The knowledge store is 0.28% of the size of the transcript store sitting next to it. A naive measurement of the parent folder reports 1.31 GiB and tells you nothing useful. One of these two you should prune without sentiment. The other you should be extremely careful with. Guess which one my tooling actually removed.

---

Failure one: the index outgrew its cap, and the overflow was silent#

The index has a size cap. Exceed it and the loader keeps the top of the file and drops the tail. No error. The session simply never sees the bottom of the list, and cannot know it is missing.

On the evening of 21 August a session opened and reported that only part of the index had loaded. Measured: 28,070 bytes against a cap of 24,576. Everything below the lighting-and-weather heading — thirteen lines carrying roughly 90 pointers — had been invisible to every session that loaded it while it sat over the line. Weather, audio, systems, characters, transport: gone from the map, still present on disk, unreachable in practice.

The cause was a rule the index was enforcing on itself. Its own header said: stay under 140 lines, merge entries onto shared lines, never truncate a line. That is a coherent rule for a cap measured in lines. The real cap is measured in bytes. Merging entries onto shared lines saves lines and saves exactly zero bytes. So every compliant edit pushed the file further over the real limit. Obedience caused the breach. Seventy-eight lines were over 150 characters by the time it broke.

The header now says the opposite, in bold, with the failure date attached — because the rule that broke it read as perfectly sensible advice right up until it didn't.

---

Failure two: the script I wrote to fix it ate the index#

The day before, a compaction pass had been launched against the same problem. Mid-session I wrote a script to hit the line target. It kept the first one or two separator-delimited segments of each line and discarded everything after them.

Line count fell 168 to 84. Entry count fell to 248. Named pointers were destroyed wholesale — a spot-check of eight names found four simply gone. The script reported success, because by its own measure it had succeeded brilliantly.

I got it back only by luck. The undamaged index had been injected into that same session's opening context, so it could be reconstructed from there. There is no version control on the knowledge store, and that absence is the only reason the recovery route was "hope a copy is in the window".

The repair then introduced fresh damage inside ten minutes: a regex shrink split a name at a hyphen and produced a pointer to nothing, and seven notes were left unreferenced entirely. Both caught, both fixed. And a verification script lied during the same pass — its name pattern demanded three hyphen-separated parts, so it announced "202 of 399 files not referenced", a false alarm large enough to bury the real, much smaller loss underneath it. What separated the artefact from the actual damage was grepping individual names by hand.

Related debris, from earlier line-shortening: three half-names left in the index — bare fragments of real filenames, pointing at nothing, reading exactly like valid entries.

---

The worst thing I found while trimming#

Three hard-won traps existed only in the index. Not in any note. The pointer list had quietly become the sole home of real knowledge — one about a node type that cannot be scripted, one about a child actor inheriting its parent's tags in a project that resolves identity by tag, one about a data file the verifiers miss because they only check two dimensions.

Any byte-trim would have deleted all three permanently, and the trim would have looked like tidying.

The rule now is: grep the topic note before trimming a trap out of the index. If the knowledge lives only in the index, that is a bug, not a memory — write it down properly first, then trim.

---

What compression actually bought#

I want to be fair to compaction, because it is not just damage control. Working out how to say a lesson in ten bytes forces you to work out what the lesson was. Several of my best warnings are the residue of a byte squeeze.

Held up:

• Keep the trigger, drop the lesson. A pointer's job is routing. "Any decal work" routes as well as three sentences explaining decals do. This was the single largest byte reclaimer.
• Bare names over titled links. Roughly four in five entries carry no title. 130 titled links had been costing 3,270 bytes of pure punctuation. Titles survive only where the title carries a number, a trap name, or a warning the filename can't.
• No counts, no stats, no totals in the index. They rot inside a week and they cost bytes forever.
• Headroom arithmetic after every edit. Print the byte count, subtract from the cap. That is how I knew there were 1,832 bytes of slack on the evening of the 21st rather than guessing.
• A two-way name diff after every index edit — index against directory listing, both directions. It catches the dropped pointer and the half-name fragment. Re-run today: clean both ways.
• State document, not journal. Current state at the top, superseded findings under a labelled history fold, each dead conclusion tagged. The origin case was a 75 KB note carrying six live self-contradictions; it came down to 43,933 bytes with every live fact intact, and it still measures exactly that a week later.

Backfired:

• Optimising for lines when the cap is bytes.
• Any destructive regex pass run without a read-back check. Twice.
• Trusting a verification script over a hand grep.

The index today is 20,884 bytes across 100 lines covering 430 notes — smaller than the day before, despite gaining a whole new topic section. About 3.7 KB of headroom, 85% full.

---

The night the whole knowledge store went, and it had nothing to do with size#

Late on 21 August the Claude Code desktop app for Windows started closing itself seconds after a session opened — and then would not reopen at all, with Windows reporting that the app could not start. Disk pressure looked plausible: the transcript store was 3,591 files and 6.17 GB, with a single 97 MB log in it.

I was offered a menu of retention windows and picked keep the last 7 days, and move to an archive rather than delete. That second choice is the only reason this story ends well.

The transcript sweep took three passes. The relative-age filter — "older than 7 days" — moved 55 files on the first pass and then matched nothing on an identical second pass over files a direct status check dated eight days earlier. A filter that silently matches nothing is indistinguishable from a job already done. The third pass abandoned relative age for an explicit wall-clock cutoff and moved 422 entries with no failures.

Then, twenty seconds later, a follow-up step went looking for orphaned attachment folders using the rule: any directory at this level with no matching transcript of the same name. The knowledge store sits at exactly that level and has no matching transcript. It was moved out whole. The output line reads: orphaned attachment dirs archived: 1.

That 1 was 426 notes.

Note what this was not. It was not age-based. Notes written an hour earlier went with the rest. The folder simply did not look like a session to a script that only knew about sessions. And sixty-eight seconds after the session ended, an empty replacement folder appeared where the store had been — so from the outside everything looked present and healthy while containing nothing at all.

The gap was 7 hours 28 minutes. At least two sessions ran inside it with no project memory. Nothing flagged it. I noticed from the outside, when a session appeared to be hunting for memory files instead of getting on with the task.

The prune also did not fix the crash. The real cause, established later and reproduced six times out of six, was the app's in-app browser preview restoring a local dev server. Two rounds of index trimming, a full restructure, a 5 GB archive, an app repair and a graphics setting were all spent on the wrong theory. And archiving to a folder on the same volume freed precisely zero bytes, so the disk-pressure theory could not have been tested by that action even if it had been right.

---

The restore, and the accidental experiment#

At 10:04 the next morning a session was midway through an unrelated rendering problem, went to read its own note on the subject, and got nothing back. Over the next ninety seconds it listed the folder (zero files), searched, found the store sitting intact in the archive, counted it — 426 files, 3,888,572 bytes — and copied it back. Restored at 10:06:13, byte-identical, nothing lost. It then read the note it had gone looking for and carried on with the original task. Twenty-four minutes later it edited that note.

The framing I started with was that the restoration tells you which memories were load-bearing. I want to be straight about how far that goes, because it is the most interesting thing here and it would be easy to oversell.

What actually happened is a whole-folder copy, not a selective regeneration. So this is not a clean experiment in which needed notes came back and unneeded ones didn't. What it is: an eight-hour window in which the system ran with no memory, and the thing that surfaced the loss was a single session needing a single note badly enough to go digging. One note out of 430 was load-bearing enough, within eight hours, to trigger a full rescue.

Read pessimistically, that is damning: 429 notes sat missing and nobody noticed. Read properly, it is the whole argument. The value of a note is not how often it is read. It is what it costs you on the day it is missing. That one note would have cost a day of re-deriving a rendering fault. Most of the other 429 cost nothing that morning, and will cost a day each, eventually, on a morning I cannot predict.

There is a second, older data point pointing the same way: after the compaction script wrecked the index, the session rebuilt it from the copy in its own context. Both recoveries were driven by need in the moment, not by any maintenance plan.

---

The pros and cons, and I am not sitting on the fence#

The case for pruning. Storage is real. An index that will not fit is not a memory, it is a liability — especially where exceeding the cap drops content silently, so the failure is invisible to the very thing it damages. And compression genuinely improves the corpus: distilling a trap down to a trigger clarifies what the trap was, and my terse warnings route better than my verbose ones ever did.

The case against age as the criterion. Age is a proxy for irrelevance and it is a bad one. A trap I hit weekly gets internalised — I barely need the note. A trap I hit once, in a subsystem I touch every couple of months, is exactly the note that saves a day, and it is exactly the note an age filter deletes first. Recency measures how recently I did that kind of work, not how much the knowledge is worth. Age-pruning a distilled knowledge base preferentially destroys the things you kept notes for in the first place.

Where I land:

• Prune the transcripts by age, freely. They are a byproduct. Mine has never held more than eight days and I have never wanted more. Two thirds of it is conversation logs; another 30% is spilled tool output, including one file sitting at exactly 64 MiB because it hit a truncation ceiling. None of that is knowledge.
• Never prune the knowledge notes by age. Not at 7 days, not at 14, not at 90. If one goes, it goes because it is wrong or superseded, and because something that has actually read it decided so.
• Compress the index hard; never compress a note merely for being long. Sixty-three notes are over 15 KB and I am leaving them. The length is the record of traps paid for in hours. Compaction is justified by contradiction, never by size.

For completeness, and because it cuts against the tidy version of this story: the 14-day automatic retention that came out of that night applies to transcripts, not notes. It has deleted zero files, because the manual sweep an hour earlier had already cut deeper than it ever will. The cleanup routine does run — it just finds nothing eligible, and will not until roughly a week from now. Configured is not the same as working, and I only know it is currently a no-op because I went and measured it.

---

Where the rot actually accumulates#

A sweep of all 430 notes found the self-contradiction backlog largely clean, but two other places rotting quietly:

Description fields. That one line is the recall surface — it is what a session reads when deciding whether to open a note at all. Nine were materially wrong. One said none of the work had ever run, about a feature verified in a live playtest. Another said 240 voice lines were unbaked, at 250 of 250 baked. A wrong description means the wrong claim gets acted on without the body ever being read.

"Still to do" lists. Nobody ever retracts one. One note listed five completed items as outstanding, including a UI element that had been deliberately abandoned for safety reasons. A stale to-do is worse than a stale fact: it commissions work that is already done, or was rejected on purpose. And my fix there was partial — I relabelled that heading as superseded rather than deleting the list, so a reader who jumps to the end still meets five finished tasks presented as pending. Honest state of play: good detector, incomplete cure.

One layer up, the same disease had hit the project's skill references. The largest had reached 222,982 bytes across 93 flat sections, so any task touching that subsystem paid 218 KB of context to read one paragraph. Four of them were split by workflow stage — 11 files became 32 — with the old filenames left in place as routers: a small table saying which successor to load. Old links still resolve, habits still work, and the byte totals corroborate that nothing was dropped, since every successor set is larger than its predecessor. The new entry point routes by symptom as well as by task, for the case where you don't yet know which subsystem is at fault. One long diagnosis was rescued in the process from inside an HTML comment, where it had been invisible for weeks.

---

What I'd tell somebody else to do#

1. Find out what your cap is measured in. Bytes, lines, tokens. Optimise the right axis, or your compliance will break the thing.
2. Measure headroom on every edit and write the number down. Guessing is how you end up 3.5 KB over.
3. Assume silent truncation. If a store can be partially loaded without erroring, a session cannot detect its own amnesia. The check has to come from outside.
4. Never run a destructive trim without a read-back diff. Names in, names out, both directions. Both of my worst losses were scripts that reported success.
5. Do not trust your verifier more than a hand grep. Mine produced a confident, wildly wrong number at the exact moment it mattered.
6. Before deleting a line from the index, grep for its content elsewhere. Anything living only in the index is knowledge masquerading as a pointer.
7. Move, never delete. The whole difference between a bad night and a lost project was one menu choice.
8. Put the knowledge store under version control. I still haven't. It is the largest outstanding hole here, and both recoveries so far ran on luck.
9. Prune by category, not by age. Byproducts by age; distilled knowledge by correctness only.
10. Watch what a script's heuristic includes, not just what it targets. "Any folder without a matching session" was correct for its intended target and catastrophic for the one folder in that location that was never a session.
11. Consent is scoped to what the question named. I approved a retention window for session history. The knowledge store was moved under that approval without ever appearing in the question.

---

The failure I watched happen while this was being written#

One more, observed live, and it is not a size problem at all.

I work with several sessions open against the same project at once — that afternoon there were about eight. They all share one index file. It has no locking, no merge, and many writers.

I measured the index at just under 24 KB. An hour later, before I had touched it again, it was under 21 KB — and a line I had added that morning was gone. It recorded that the project had switched email providers that day. The underlying note was intact and still said the right thing; only the pointer to it had vanished.

I cannot tell you which mechanism did it. Another session may have compacted the index deliberately and judged that line expendable, which is defensible. Or two sessions read the file, both edited, and the last writer won. Both look identical afterwards, because neither leaves a trace. What I can tell you is that the result is the same either way: the index quietly stopped pointing at something true, and nothing anywhere reported a problem.

That is worse than the overflow, because overflow at least has a threshold you can measure and defend against. This has neither. And it explains the symptom that started this whole week — recent things being forgotten — at least as well as the byte cap does. Recent facts are the ones that exist in exactly one place, held by exactly one session, not yet reinforced anywhere else. They are the cheapest thing for a compactor to drop and the easiest thing for a concurrent write to lose.

The mitigation I have not built yet, and should: stop hand-editing the shared index at all, and generate it from the notes. A file that is derived cannot lose a fact that a note still holds. Until then I re-check the index after any session where several were running, which is a discipline, and disciplines fail.

What I don't know#

• How long the index sat over its cap before I noticed. It was already 27 KB the previous evening, so earlier sessions were probably loading it truncated too. No direct evidence either way.
• Whether the compaction script lost any entry text permanently. The post-repair check verified the pointer set, not the wording, and no before-and-after text diff survives.
• Whether anyone would have noticed the missing knowledge store at all if that one session hadn't happened to need that one note.
• Whether the skill split is genuinely verbatim. The provenance lines say so and the byte totals are consistent with it, but I have not diffed the backups section by section.
• What happens at the ceiling. At roughly 28 bytes a pointer, the index supports maybe 450 to 500 notes before the cap forces a structural split instead of another trim. I am at 430. The plan is "split it"; there is no design for that split yet.
• Timestamps are no help in reconstructing any of this. 82% of the notes share a single modification date from a bulk rewrite, and 218 archived transcripts share one identical millisecond stamp from some earlier bulk operation. The corpus cannot report its own maintenance history; the narrative survives only because somebody wrote it down. Which is, more or less, the entire point.