Stop dream consolidation from destroying memories

Directory.Build.props

@@ -9,7 +9,7 @@
     <!-- Version can be overridden from the command line: -p:Version=0.3.1
          AssemblyVersion and FileVersion are derived automatically by the SDK
          (prerelease suffixes like -beta001 are stripped for assembly versions). -->
-<Version>0.14.6</Version>
+<Version>0.14.10</Version>
   </PropertyGroup>
 
   <!-- NuGet package metadata (shared across all packable projects) -->

deploy/helm/rockbot/templates/agent/deployment.yaml

@@ -76,6 +76,13 @@ spec:
                 cp /app/agent/llm-pricing.json /data/agent/llm-pricing.json
               fi
 
+              # merge-coverage-vocabulary.json — per-deployment word list for the dream
+              # merge-coverage safeguard. Copy no-clobber so operator tuning survives upgrades.
+              if [ -f /app/agent/merge-coverage-vocabulary.json ] && [ ! -s /data/agent/merge-coverage-vocabulary.json ]; then
+                echo "  Copying default merge-coverage-vocabulary.json"
+                cp /app/agent/merge-coverage-vocabulary.json /data/agent/merge-coverage-vocabulary.json
+              fi
+
               # Per-model behavior prompt files — copy per-file so users can customise
               # individual prompts without losing others on upgrade
               for model_dir in /app/model-behaviors/*/; do

docs/dream-service.md

@@ -68,22 +68,56 @@ are not swept.)
 
 **Enabled/disabled by:** `DreamOptions.LogRetentionEnabled` (default `true`).
 
+### Pass 0b — Archive purge
+
+Hard-deletes memory entries archived longer ago than `DreamOptions.MemoryArchiveRetention`
+(default 90 days). Runs before consolidation so the retention window is measured from the
+archive event rather than from whatever the current cycle is about to archive.
+
+Requires the store to implement `IArchivedMemoryMaintenance`; with a store that does not,
+`ArchiveAsync` falls back to a hard delete and there is nothing to purge. A non-positive
+retention keeps archived entries forever.
+
 ### Pass 1 — Memory consolidation
 
-**Input:** all long-term memory entries (up to 1000) + recent feedback signals (last 7 days,
-up to 50). Each entry is rendered with its temporal context — `first=` (CreatedAt),
+**Input:** a *gated subset* of long-term memory entries + recent feedback signals (last 7
+days, up to 50). Each entry is rendered with its temporal context — `first=` (CreatedAt),
 `last=` (LastSeenAt), `reinforced=N×` (ReinforcementCount), and `subject=...` when
 subject-time metadata is present — so the LLM can reason about whether similar-sounding
 entries describe the same durable fact or distinct moments.
 
+**Candidate gating — what the LLM is allowed to see.** An entry becomes eligible only if it
+is (a) new or changed since its last review, or (b) part of a near-duplicate cluster. Anything
+else is withheld and therefore cannot be archived this cycle. Enforcement is in code, not just
+in the prompt: the source lookup used for merge arithmetic is keyed on the eligible set, so an
+ID the model invents or remembers from a previous cycle resolves to nothing.
+
+This exists because exposure compounds. Handing over the whole corpus every cycle means every
+entry is re-tried for deletion every cycle; at the default twice-daily cadence, a
+one-in-a-thousand misjudgement per entry per cycle loses roughly half the corpus in a year.
+Gating makes it roughly one decision per entry per content change.
+
+- Review state is a content fingerprint (`consolidationReviewedHash` in entry metadata), so
+  any write path — reinforcement, a tool edit, a prior merge — re-opens an entry for review,
+  while importance decay (which changes score and `UpdatedAt`, not content) does not.
+- Clustering comes from `IMemoryDuplicateCandidates` on the store: cosine over embeddings
+  where available, Jaccard over content tokens otherwise, so BM25-only deployments still
+  deduplicate. Controlled by `Dream:ConsolidationSimilarityThreshold` (default 0.88) and
+  `Dream:ConsolidationMaxClusterSize` (default 3). The cluster cap bounds *eligibility*, not
+  merge size — the model may propose a merge over any subset of what it is shown, and in
+  practice does produce merges with more sources than this value. Large merges are constrained
+  by the coverage check below rather than by a count.
+- If the near-duplicate scan fails, the pass degrades to unreviewed-only rather than falling
+  back to the whole corpus.
+
 **What the LLM does:**
 - Merges duplicate and near-duplicate entries into single improved entries, including
   widely-separated observations of the same durable fact (treating them as reinforcement,
   not novelty)
 - Preserves topically-similar entries that describe distinct real-world moments (different
   trips, meetings, incidents) — especially when subject-time differs sharply
 - Refines categories (e.g. promotes `general` entries to more specific categories)
-- Deletes noisy, low-value, or fully superseded entries
+- Flags noisy, low-value, or fully superseded entries for removal
 - Mines `Correction` feedback for anti-patterns and writes them to `anti-patterns/{domain}`
 
 **Temporal-field arithmetic on merge (computed by the host, not the LLM):**
@@ -98,9 +132,71 @@ This division keeps the LLM focused on *what to merge* and prevents dream housek
 stamping every reprocessed entry as "just observed." See `dream.md` for the Temporal merging
 rules the LLM is given.
 
-**Exhaustive deletion contract:** The union of explicit `toDelete` IDs and all `sourceIds`
-referenced in merged entries are deleted. This prevents orphaned source entries when the LLM
-omits IDs from `toDelete` but lists them in `sourceIds`.
+**Removals are archived, never deleted.** Consolidation calls `ILongTermMemory.ArchiveAsync`,
+which hides an entry from search while keeping it on disk and retrievable by ID. The archive
+purge pass hard-deletes entries archived longer ago than `Dream:MemoryArchiveRetention`
+(default 90 days), so a wrong merge or a wrong "ephemeral" call costs recall for a while
+rather than costing the fact. Every archive is logged at Information with the entry's content
+inline, which is what makes a bad cycle reviewable without restoring a volume backup.
+`IArchivedMemoryMaintenance.RestoreAsync` puts an entry back.
+
+**Merge coverage check.** Before a merge is applied, the specifics in its sources — proper
+nouns, acronyms, and multi-digit numbers — must all appear in the merged text. If any is
+missing the merge is rejected outright: nothing is saved and the sources are left alone. This
+is what catches the characteristic failure, a plausible-reading merge that keeps the
+machine-readable half of an entry and quietly drops a name or a date.
+
+**Vocabulary is per-deployment.** Which capitalized words count as ordinary language rather
+than as detail is not portable between agents, so it lives in
+`merge-coverage-vocabulary.json` on the agent profile volume (next to `tier-selector.json`),
+re-read at the top of every cycle:
+
+```json
+{
+  "extraCommonWords":    ["briefing", "triage"],
+  "alwaysSpecificWords": ["May", "Will", "Rose"]
+}
+```
+
+`extraCommonWords` suppresses domain noise. `alwaysSpecificWords` reclaims words from the
+built-in generic-English baseline and takes precedence over it — this matters most for agents
+whose people or characters collide with ordinary English. The baseline contains `may`, `will`,
+`some`, `first` and `last`, so a storytelling agent with a character named **May** or **Will**
+must list them here or those names carry no coverage protection at all. A malformed file falls
+back to the baseline with a warning; coverage checking is never disabled by bad config.
+
+The check is deliberately biased toward rejection, because the costs are asymmetric: a false
+rejection leaves a duplicate pair alive for another cycle, while a false acceptance destroys
+the only record of how a fact was worded. Measured against a real 148-entry corpus, it rejects
+0% of merges that preserve all source content and catches 83% of merges that drop a source
+outright (the remainder being pairs where one source's specifics are a strict subset of the
+other's — genuinely redundant). Known conservative false positives include 12h→24h clock
+reformatting.
+
+**High-value pruning floor.** Entries at or above `Dream:PruningProtectionImportance`
+(default 0.80) or `Dream:PruningProtectionReinforcementCount` (default 5) can be merged, but
+are never archived as standalone ephemeral. Merging preserves content and is covered by the
+check above; ephemeral pruning discards a fact with nothing in its place, which is not
+something to do on one model's say-so to an entry the agent has re-observed dozens of times.
+This is a deterministic floor precisely because the prompt-level version did not hold —
+`dream.md` already said reinforcement signals importance, and a live corpus still lost entries
+reinforced 214, 106 and 80 times.
+
+**Provenance.** Merged entries carry `mergedFrom` (source IDs) and `mergedAt` in metadata.
+Source text is not duplicated: the sources are archived rather than deleted, so those IDs
+resolve via `GetAsync` for the retention window. Metadata is not part of the search surface,
+so this does not affect ranking. After the purge the IDs dangle by design.
+
+**Ordering — replacement first, then retirement.** Merged entries are saved *before* their
+sources are archived, and only sources belonging to a merge that actually persisted are
+retired. A `toSave` entry with blank content is skipped with a warning and its sources are
+kept. (The previous order deleted everything up front and saved afterwards, so a skipped or
+failed save destroyed the sources outright.)
+
+**Exhaustive-removal contract:** The union of explicit `toDelete` IDs and all `sourceIds`
+referenced in *persisted* merged entries is archived. This prevents orphaned source entries
+when the LLM omits IDs from `toDelete` but lists them in `sourceIds`. IDs outside the eligible
+set are ignored.
 
 **Episode reinforcement:** When a new session revisits an existing episodic memory (found by
 the episode extraction pass), the existing entry is updated with `LastSeenAt = now` and

docs/memory.md

@@ -387,21 +387,28 @@ The dream service runs two memory-related passes:
 
 ### Pass 1 — Memory consolidation
 
-Reviews all long-term memory entries for duplicates, near-duplicates, and outdated content.
+Reviews a gated subset of long-term memory entries for duplicates, near-duplicates, and
+outdated content.
 
 **Inputs provided to the LLM:**
-- All memory entries (up to 1000), numbered with ID, category, tags, content, and temporal
-  context (`first=`, `last=`, `reinforced=N×`, and `subject=...` when subject-time metadata
-  is present)
+- Only *eligible* entries — those new or changed since their last review, plus those in a
+  near-duplicate cluster — numbered with ID, category, tags, content, and temporal context
+  (`first=`, `last=`, `reinforced=N×`, and `subject=...` when subject-time metadata is
+  present). Everything else is withheld and cannot be touched this cycle.
 - Recent feedback signals (last 7 days, up to 50) for quality context
 
 **What the LLM can do:**
 1. Merge duplicate or near-duplicate entries — even when widely separated in time, treating
    them as reinforcement rather than novelty
 2. Refine categories and tags
-3. Delete noisy or redundant entries
+3. Flag noisy or redundant entries for archiving
 4. Write `anti-patterns/{domain}` entries from Correction feedback
 
+**Why gating exists:** without it, the whole corpus is re-offered for deletion on every cycle,
+so per-entry survival compounds against you — at twice a day, a one-in-a-thousand misjudgement
+per entry per cycle loses about half the corpus in a year. See `dream-service.md` for the
+eligibility rules and tuning knobs.
+
 **Temporal arithmetic on merge (computed by the host, not the LLM):**
 
 - `CreatedAt` = `min(sources.CreatedAt)` — earliest first-seen preserved
@@ -417,8 +424,26 @@ This split keeps the LLM focused on *what to merge* while the host guarantees co
 temporal arithmetic and prevents the dream cycle from inadvertently stamping every
 reprocessed entry as "just seen today."
 
-**Exhaustive deletion contract:** The union of explicit `toDelete` IDs and all `sourceIds` from
-merged entries are deleted — preventing orphaned source entries even if the LLM omits some IDs.
+**Two safeguards constrain what a pass may do:**
+
+- *Merge coverage* — every proper noun, acronym and multi-digit number in a merge's sources
+  must survive into the merged text, or the merge is rejected and the sources are kept.
+- *High-value floor* — entries at or above `Dream:PruningProtectionImportance` (0.80) or
+  `Dream:PruningProtectionReinforcementCount` (5) may be merged but never pruned outright.
+
+Both are deterministic rather than prompt guidance, because the prompt-level versions were
+already present and did not hold. Merged entries record `mergedFrom` / `mergedAt` in metadata
+(metadata is not part of the search surface). See `dream-service.md` for measured behaviour.
+
+**Removals are archived, not deleted.** Consolidation calls `ArchiveAsync`, which hides an
+entry from search but keeps it on disk and retrievable by ID; a separate purge pass
+hard-deletes archived entries after `Dream:MemoryArchiveRetention` (default 90 days), and
+`IArchivedMemoryMaintenance.RestoreAsync` brings one back. Merged entries are saved before
+their sources are archived, and only sources whose replacement actually persisted are retired.
+
+**Exhaustive-removal contract:** The union of explicit `toDelete` IDs and all `sourceIds` from
+persisted merged entries is archived — preventing orphaned source entries even if the LLM
+omits some IDs. IDs outside the eligible set are ignored.
 
 ### Pass 2 — Preference inference
 

src/RockBot.Agent/RockBot.Agent.csproj

@@ -38,6 +38,9 @@
     <None Include="agent\llm-pricing.json" CopyToOutputDirectory="PreserveNewest">
       <Link>agent\llm-pricing.json</Link>
     </None>
+    <None Include="agent\merge-coverage-vocabulary.json" CopyToOutputDirectory="PreserveNewest">
+      <Link>agent\merge-coverage-vocabulary.json</Link>
+    </None>
   </ItemGroup>
 
   <ItemGroup>

src/RockBot.Agent/agent/dream.md

@@ -4,7 +4,14 @@ You are a memory consolidation assistant performing a maintenance pass over an a
 
 ## Your task
 
-You will receive a numbered list of ALL current memory entries, each with an ID, category, tags, and content. Review them and:
+You will receive a numbered list of **some** of the agent's memory entries — each with an ID, category, tags, and content. This is a filtered working set, not the whole corpus: it contains entries that are new or changed since the last pass, plus entries that look like near-duplicates of one another. Everything else has already been reviewed, has not changed since, and is deliberately withheld.
+
+Two consequences:
+
+- **Only ever reference IDs from the list you were given.** IDs you remember from a previous pass, or infer, are rejected. A fact you cannot see is not missing — it is withheld, and it is fine.
+- **Do not try to tidy the corpus as a whole.** You are not seeing the whole corpus. Judge only what is in front of you, on its own merits.
+
+Review the listed entries and:
 
 1. **Re-evaluate importance scores** — each entry has a current importance score (0.0–1.0). Adjust scores based on:
    - How central the fact is to the agent's primary work and user's goals
@@ -79,6 +86,7 @@ whether similar entries describe the same durable fact or distinct facts.
 ## Critical rules
 
 - **Exhaustive deletion — this is the most important rule**: Every source entry you are replacing with a merged entry MUST appear in `toDelete`. If you produce one merged entry from sources A, B, and C, then A, B, and C ALL go in `toDelete`. No source survives a merge. The presence of an ID in `sourceIds` is a commitment to delete it — put it in `toDelete` too.
+- **A merged entry must carry every specific its sources carried**: names of people, places, and organisations, dates, numbers, and identifiers. Losing "Rocky also appears in travel data as Rockford Duane Lhotka" while keeping the surrounding prose is a failed merge, not a tidier one. If a merge cannot hold all the specifics at a reasonable length, do not merge — leave the sources alone.
 - **No orphaned sources**: After your pass, there must be no entry whose content is fully captured by a new entry you saved. If a fact is in your merged output, its source is deleted.
 - **Conservative on merging**: When in doubt whether two entries are truly duplicates, keep both. But when you do merge, delete ALL sources completely.
 - **Never delete without replacement**: Do not delete a unique fact that has no equivalent in your output. Ephemeral entries are the only exception.

src/RockBot.Agent/agent/merge-coverage-vocabulary.json

@@ -0,0 +1,26 @@
+{
+  // Merge-coverage vocabulary — tunes the safeguard that stops dream consolidation from
+  // dropping details when it merges memories.
+  //
+  // The check requires every proper noun, acronym and multi-digit number in a merge's sources
+  // to survive into the merged text. Words listed as "common" are exempt from that requirement.
+  // A built-in generic-English list is always applied; this file layers on top of it.
+  //
+  // Re-read at the top of every dream cycle — no restart needed. If this file is malformed the
+  // built-in list is used and a warning is logged; coverage checking is never disabled.
+
+  // Words to ALSO ignore. Add domain noise that keeps triggering rejections you judge to be
+  // false. Be conservative: a false rejection only costs a duplicate surviving another cycle,
+  // whereas suppressing a real name lets it be deleted silently. Do not add a word that could
+  // ever name something — "Personal", "Class" and "Benefit" look generic but carry meaning in
+  // "OneDrive Personal", "Blazor Online Class" and "MVP Azure Extended Benefit".
+  "extraCommonWords": [],
+
+  // Words to ALWAYS treat as specifics, overriding the built-in list.
+  //
+  // This matters most for agents whose people or characters collide with ordinary English.
+  // The built-in list contains "may", "will", "some", "first", "last", "new" and similar, so a
+  // storytelling agent with a character named May, Will, Rose, Grace or Hope MUST list them
+  // here — otherwise those names carry no coverage protection and a merge can drop them.
+  "alwaysSpecificWords": []
+}