Stop dream consolidation from destroying memories

[rockfordlhotka]

The problem

Dream consolidation handed the entire memory corpus to the LLM every cycle with an open licence to delete, then hard-deleted whatever came back — File.Delete, no tombstone, and the only trace was a Debug log line containing the entry ID, not its content. So a loss was both unrecoverable and uninvestigable.

Exposure compounds. At the default twice-daily cadence each entry faces 730 deletion decisions a year:

Per-entry, per-cycle survival	Survives 1 year
99.99%	93%
99.9%	48%
99.5%	3%

A one-in-a-thousand misjudgement loses half the corpus in a year. No directive is good enough to beat a 730×-repeated gamble.

Measured, not theorised

Restored a Longhorn backup of a live agent's PVC and diffed it against the running store. Over 3.5 days: 148 → 109 entries, −26%. Named things that vanished corpus-wide: Duane, Trish, Roberts, PWOP Productions, Las Vegas, Tripit, BlazorBook, BlazorHol.

Importance and reinforcement provided zero protection. Among entries whose content survived nowhere:

agent-knowledge/self-model          imp=0.99  reinforced=214x
anti-patterns/routing               imp=1.00  reinforced=106x
agent-knowledge/infrastructure      imp=0.99  reinforced= 80x
user-preferences/identity           imp=0.99  reinforced= 73x
project-context/blazor-online-class imp=0.97  reinforced= 41x

dream.md already told the model reinforcement signals importance. It deleted them anyway. That is why every safeguard here is deterministic rather than prompt text.

The merge failure mode is subtle: user-preferences/identity's successor kept the machine-readable account map and silently dropped "Rocky Lhotka also appears in travel and calendar data as Rockford Duane Lhotka." The result reads fine. Nothing flagged it.

Changes

Archive instead of delete. MemoryEntry gains ArchivedAt/ArchiveReason; ILongTermMemory.ArchiveAsync hides an entry from search while keeping it on disk and retrievable by ID. Default interface method delegating to DeleteAsync, so existing fakes compile untouched. Recovery/retention sit on an optional IArchivedMemoryMaintenance; a purge pass hard-deletes after Dream:MemoryArchiveRetention (90d). Archives log at Information with content inline.

Candidate gating. An entry is eligible only if new/changed since its last review, or in a near-duplicate cluster. Review state is a content fingerprint, not a timestamp — importance decay rewrites score and UpdatedAt but not content, so decayed entries don't leak back through the gate. Clustering sits behind IMemoryDuplicateCandidates: cosine where embeddings exist, Jaccard otherwise, so BM25-only deployments still dedupe. Enforced in code — merge arithmetic is keyed on the eligible set, so an ID outside it resolves to nothing. A clustering failure degrades to unreviewed-only, never back to the whole corpus.

Merge coverage check. Proper nouns, acronyms and multi-digit numbers in a merge's sources must appear in the merged text, or the merge is rejected and the sources are left alone. Deliberately biased toward rejection: a false rejection leaves a duplicate alive one more cycle; a false acceptance destroys the only record of how a fact was worded.

Validated against the real 148-entry corpus:

• 0 self-coverage failures
• 0/300 false rejections on content-preserving merges
• 250/300 (83%) of detail-dropping merges caught

The 17% missed are pairs where one source's specifics are a strict subset of the other's — genuinely redundant. That empirical pass caught two false-rejection bugs the unit tests missed: possessives (Rocky's vs Rocky, 27 occurrences) and bare single digits (top 3 → top three).

High-value pruning floor. Entries at or above Dream:PruningProtectionImportance (0.80) or Dream:PruningProtectionReinforcementCount (5) can be merged — content survives, coverage-checked — but are never archived as standalone ephemeral.

Provenance. mergedFrom/mergedAt on merged entries. Source text is not duplicated: sources are archived, so IDs resolve for the retention window. Metadata is outside the search surface, so ranking is unaffected.

Latent data-loss bug fixed. The pass deleted all sources up front, then saved. A toSave with blank content hit a continue — after its sources were already gone. Any throw between the loops did the same at scale. Merged entries are now saved first, and only sources whose replacement persisted are retired.

Honest scope

Gating alone would not have prevented most of the observed losses — high-value entries are frequently reinforced and have near-duplicate siblings, which is exactly what keeps them eligible. For those, the coverage check and the pruning floor are the prevention, and archiving is the safety net.

Still open (follow-up, all pre-existing and now much less dangerous):

• dream.md still licenses deletion at near-zero importance
• Retrieval doesn't bump LastSeenAt, so a memory used weekly still decays
• No category exemptions for user-preferences/**
• Consolidation still caps at MaxResults: 1000

Testing

dotnet build RockBot.slnx clean; all 19 test projects pass. 35 new tests across MemoryArchiveTests, ConsolidationCandidateGatingTests and MergeCoverageTests — including a regression test built from the actual Rockford Duane merge that lost the name.

Deployment note

Consolidation is currently disabled on the live agent (Dream__MemoryConsolidationEnabled=false) to stop the bleeding. After deploying, re-enable and watch one cycle for the new Information lines: reviewing N of M entries, consolidation safeguards fired, refused to prune. If merge rejections run high, ConsolidationSimilarityThreshold is the dial.

🤖 Generated with Claude Code

[rockfordlhotka]

Live validation on the production agent (0.14.7 → 0.14.8)

Deployed to the cluster and watched two real dream cycles against a 275-entry corpus. The first run found a defect the unit tests could not.

Cycle 1 — 0.14.7, cold start

consolidation reviewing 251 of 251 entries (0 withheld)
rejected merge of [9 sources] — dropped 1 specific(s): Xebia
rejected merge of [4 sources] — dropped 7 specific(s): 0.77536025, 5.67, 8.33, Adding, Candidate, Flagged, Validated
safeguards fired — 2 merge(s) rejected, 0 protected from pruning
dream cycle complete — 17 deleted, 19 saved

0 withheld is expected on a cold start — no entry carried a review stamp yet.

Bug found: the rejected merges' sources were archived anyway, as flagged ephemeral. dream.md requires every sourceId to also appear in toDelete, so rejecting the merge skipped the save but left those IDs in the standalone-removal list. The rejection achieved nothing, and was strictly worse than allowing the lossy merge — the sources went with no replacement at all. 13 entries, 13 of 17 archives that cycle.

The unit tests test FindMissingSpecifics in isolation; the defect lives in the interaction between two loops in the caller and only triggers on real LLM output with that overlap shape.

All 13 recovered by clearing their archive stamps — which is the archive tier doing its job. Under the old code they would have been destroyed silently.

Fixed in 403d0d6, with a regression test reproducing the production shape.

Cycle 2 — 0.14.8, fix applied

consolidation reviewing 167 of 274 entries (107 withheld as reviewed-and-unchanged)
refused to prune 167159126119 (active-plans/..., importance=0.85) — above the high-value floor
safeguards fired — 8 merge(s) rejected for dropping specifics, 1 entry(s) protected
dream cycle complete — 52 deleted, 12 saved

Signal	Result
Gating	107 of 274 withheld (39%) — the ratchet biting
Rejected-merge sources archived	0 (was 13)
Archives with a replacement	51 of 52
Ephemeral archives (no replacement)	1
Hard deletions	0

Merge quality. A 13-source Allen Conway merge passed the coverage check and kept every specific — Allen.Conway@xebia.com, atconway@bellsouth.net, +1 386-299-4605, Principal Consultant, VS Live, Xebia — with reinforcement summed to 58 and all 13 source IDs in mergedFrom. That is the shape that previously lost Rockford Duane.

Coverage-check precision. 8 rejections; 6 caught a genuinely dropped specific (Bluesky/Mastodon, Atlanta/ATL, 2026-08-03/August, IMAP/LLC, several band and person names, PNG). 2 were driven by ordinary words opening a sentence (IDs, Downloading, Personal). False positives only cost a duplicate surviving another cycle, so this errs the right way, but the stoplist could still be trimmed — noted as follow-up rather than fixed here.

Caveat

Two cycles on one corpus is not proof of general calibration. Cycle 1 already showed a shape that unit tests missed. Worth watching several more cycles before treating the safeguards as settled.

[rockfordlhotka]

Four more live cycles (0.14.8 → 0.14.10), plus a design fix

Continued running against the production agent. Six cycles total now; the previous comment covered the first two.

Cycle table

Cycle	Version	Reviewed / withheld	Archived	— with replacement	— ephemeral	Saved	Rejected	Protected	Hard deletes
1	0.14.7	251 of 251 (0%)	17	4	13 †	19	2	0	0
2	0.14.8	167 of 274 (39%)	52	51	1	12	8	0	0
3	0.14.8	128 of 239 (46%)	31	30	1	8	4	0	0
4	0.14.9	99 of 222 (55%)	4	4	0	2	4	0	0
5	0.14.9	100 of 228 (56%)	5	5	0	1	3	0	0
6	0.14.10	113 of 241 (53%)	13	11	2	4	3	1	0

† the bug fixed in 403d0d6 — rejected merges' sources were reaching the ephemeral path.

Gating converged. Withholding settled in the low-to-mid 50s. The corpus stopped shrinking: live entries went 275 → 240 → 223 → 229 → 242 → 236, i.e. it consumed the duplicate backlog that accumulated while consolidation was off, then stabilised with mining outpacing consolidation. The early drop was transient, not erosion.

Zero hard deletions across all six cycles. Every removal is recoverable for 90 days.

The safeguards caught real losses

Cycle 3 — a merge that would have collapsed six entries into one narrow technical note:

rejected merge of [6 sources] — dropped 28 specific(s): 12:00, 12:30, 187473972815, 19, 1:30,
  2026, August, Azure, Benefit, Blazor, Carl, Check, Class, Code, Communications, Extended,
  Franklin, Friday, Lacey, Matt, Microsoft, MVP, Online, PGI, PM, Power, RV, Social

Carl Franklin, Matt and Lacey — three more people, about to go the way of Trish Roberts.

Cycle 6 — the high-value floor fired for the first time:

refused to prune 8ef07d496982 (agent-knowledge/infrastructure, importance=0.99, reinforced=80x)
  — above the high-value floor

That is one of the exact entries from the original forensic table (agent-knowledge/infrastructure imp=0.99 reinforced=80x). The model proposed discarding it again, with nothing in its place. A deterministic floor refused. Under the old code it would have been destroyed a second time.

Design fix: the coverage stoplist was hardcoded (a503545)

Raised in review, and correct: a word list is vocabulary, and vocabulary is deployment-specific. The failure was worse in the storytelling direction — the built-in list contains may, will, some, first, last, so a narrative agent with a character named May, Will or Rose would have had those names silently stripped of coverage protection. That is precisely the population the safeguard exists to protect.

Vocabulary now loads from merge-coverage-vocabulary.json on the agent profile volume, beside tier-selector.json, re-read at the top of every cycle:

{ "extraCommonWords": ["briefing"], "alwaysSpecificWords": ["May", "Will", "Rose"] }

alwaysSpecificWords wins over the built-in list. The generic-English baseline stays in code as the default, so behaviour is unchanged with no file present, and a malformed file falls back to the baseline with a warning — bad config must never silently disable coverage checking.

Delivering it exposed two further gaps, both silent (51f53c3): the agent csproj copies agent\**\*.md plus individually-listed JSON, and the chart's init container seeds *.md plus individually-listed JSON. A .json added to agent/ matched neither, so no deployment would ever have received the file. It degraded gracefully to the baseline, which is exactly why it would not have been noticed.

Correction: ConsolidationMaxClusterSize (23bda77)

I had documented it as capping merge fan-in. It does not — it bounds which entries are shown; the model may then merge any subset. Production ran a 13-source merge under the default of 3.

The behaviour is right and the docs were wrong: an arbitrary fan-in cap would have blocked that 13-source merge, which was excellent — it consolidated 13 fragmentary entries about one colleague while preserving both email addresses, the phone number, the job title and every other specific, summing reinforcement to 58. The same cycle rejected a 6-source merge that dropped 28 specifics. The coverage check is the better constraint because it judges whether detail survived rather than guessing from a count. Docs corrected in all four places that carried the claim.

Observability (c82d478)

The vocabulary load logged at Debug, so a live cycle produced only circumstantial evidence about whether an override was in effect — for a file that decides what a merge may discard. Now at Information, once per cycle:

DreamService: merge-coverage vocabulary from /data/agent/merge-coverage-vocabulary.json
  — 206 common words, 0 reclaimed as specifics

Built-in baseline is 205 unique words, so the 206 is direct proof the file was read. Also confirmed the customised file survives an image upgrade untouched (init container no-clobber).

Caveats

• Cycle 6 archived more than 4 and 5 did, so this is noise around a steady state, not a clean monotonic curve. One more cycle would establish the resting rate.
• Ephemeral archives (removal with no replacement) are the only category that discards outright — 2 in cycle 6, both low-value. That is the number to watch.
• The Rocky entry in rockbot's vocabulary file is PVC-only and deliberately not shipped in the repo default; a repo default would impose one deployment's vocabulary on all of them.
• muse needs an alwaysSpecificWords list before consolidation is ever enabled there. It is disabled today so nothing is at risk, but turning it on without that file is the character-name failure above, live.