fix(postgres): prevent reconciliation storage bloat

PG-PROFILE-51-REPORT.md

@@ -167,8 +167,8 @@ tail:
 ## Scope counts and comparison baseline
 
 `scripts/run-tests.ts` discovers 1,034 profile files. The template equivalence
-registry is a different measure: 122 eligible plus 26 cold-required files,
-for **148** total. No current tracked source contains a 170-file claim.
+registry is a different measure: 122 eligible plus 30 cold-required files,
+for **152** total. No current tracked source contains a 170-file claim.
 
 The sibling report supplies only a memory-default cap-2 baseline (10,252
 assertions, 9,690 passed, 209 failed, 353 skipped; about 10 minutes). It is not

ai/research/260903-postgres-bloat-maintenance.md

@@ -0,0 +1,24 @@
+# PostgreSQL bloat maintenance
+
+Date: 2026-09-03
+
+## Question
+
+How should the reference implementation prevent high-churn PostgreSQL tables from accumulating physical bloat, and how can it reclaim derived-index space without stopping writers?
+
+## Findings
+
+- PostgreSQL supports table-specific autovacuum parameters with `ALTER TABLE ... SET (...)`; the same reloptions can address the table's TOAST relation with the `toast.` prefix. The relevant vacuum trigger is a threshold plus a scale-factor term, so low scale factors alone do not protect a table that has only a few very large JSONB or BYTEA rows.
+- `VACUUM (ANALYZE)` cannot run inside a transaction block. `REINDEX ... CONCURRENTLY` also cannot run inside a transaction block, so the maintenance runner must borrow a direct session rather than use the transaction helper.
+- A concurrent reindex allows normal writes but has stricter operational constraints. The runner therefore rebuilds only an explicit, static allowlist of derived indexes after observed dead-tuple pressure crosses a conservative threshold.
+
+## Decision
+
+The source-level fix is write elision: repeated structural-equality record payloads do not mutate `records` or append `record_changes`; content-addressed `blobs` already does the same for binary payloads. The implementation also avoids no-op metadata updates on that record path. These properties are separately proven with physical PostgreSQL relation statistics. Per-table heap and TOAST autovacuum reloptions for `records`, `record_changes`, `blobs`, and `spine_events`, plus the default 01:00-05:00 UTC maintenance job, are recovery safety nets only; they do not make a rewriting source path correct. The job vacuums known heavy tables and considers concurrent rebuild only for a static search-index allowlist. A durable database receipt claims each UTC window so a restart or another replica does not repeat an expensive partial pass. Set `PDPP_POSTGRES_DERIVED_INDEX_MAINTENANCE_WINDOW=disabled` to opt out or set another UTC window. A health receipt shows the last completed local job outcome.
+
+## Sources
+
+- PostgreSQL: [Routine Vacuuming](https://www.postgresql.org/docs/current/routine-vacuuming.html)
+- PostgreSQL: [Automatic Vacuuming](https://www.postgresql.org/docs/current/runtime-config-vacuum.html)
+- PostgreSQL: [REINDEX](https://www.postgresql.org/docs/current/sql-reindex.html)
+- PostgreSQL: [CREATE TABLE storage parameters](https://www.postgresql.org/docs/current/sql-createtable.html)

ai/research/INDEX.md

@@ -0,0 +1,3 @@
+# Research index
+
+- [260903 PostgreSQL bloat maintenance](260903-postgres-bloat-maintenance.md) — table-level autovacuum options and concurrent reindex constraints for the 2026-09-03 bloat repair.

inbox/devspecs-feedback.md

@@ -0,0 +1,9 @@
+# devspecs feedback
+
+## 2026-09-03 — DB bloat repair
+
+`ds task "fix PostgreSQL storage bloat" --slice ...` waited at “Task index preflight: waiting for another index update” for more than 30 seconds and never produced a task slice. The command gave no owner, timeout, or recovery action, so I continued with the repository brief and targeted tests. A bounded wait plus a suggested retry/status command would make this easier to use during incident work.
+
+## 2026-09-09 — Concurrent blob cleanup repair
+
+`ds recent` completed in about six seconds and identified the existing reconciliation-bloat change and its files. `ds task "preserve shared PostgreSQL blobs during concurrent connector deletion" --quick` discovered 3,537 files, then stayed at “extracting and indexing artifacts” for over four minutes without producing a task. I stopped that invocation and continued from the lane brief and PostgreSQL regression tests. A quick task still needs a bounded path that can use known file paths without a full index.

reference-implementation/scripts/postgres-template-eligibility.ts

@@ -170,9 +170,12 @@ export const POSTGRES_TEMPLATE_COLD_REQUIRED_FILES: readonly string[] = [
   "test/connector-summary-source-revision.test.ts",
   "test/device-ingest-reservation-migration.test.ts",
   "test/polyfill-manifest-reconcile-invalidation-postgres.test.ts",
+  "test/postgres-blob-publication-conflict.test.ts",
   "test/postgres-boot-migration-resume.test.ts",
   "test/postgres-bootstrap-deadlock-retry.test.ts",
+  "test/postgres-derived-index-maintenance.test.ts",
   "test/postgres-hnsw-postlisten.test.ts",
+  "test/postgres-orphan-blob-reclaim.test.ts",
   "test/postgres-record-index-bootstrap.test.ts",
   "test/postgres-record-index-idempotency-oracle.test.ts",
   "test/postgres-record-index-repair-oracle.test.ts",
@@ -183,6 +186,7 @@ export const POSTGRES_TEMPLATE_COLD_REQUIRED_FILES: readonly string[] = [
   "test/run-history-duplicate-run-id-identity.test.ts",
   "test/run-history-interrupted-migration-reconciliation.test.ts",
   "test/run-history-writer-authority.test.ts",
+  "test/semantic-index-skip-unchanged-postgres.test.ts",
   "test/spine-events-connector-instance-id-backfill.test.ts",
   "test/spine-source-boot-backfill.test.ts",
 ];

reference-implementation/server/backup-table-policy.ts

@@ -233,6 +233,11 @@ export const BACKUP_TABLE_INVENTORY: Record<string, BackupTableInventoryEntry> =
     classification: "backup_required",
     reason: "Pending consent transactions must not be silently dropped by a coherent restore.",
   },
+  postgres_derived_index_maintenance_receipts: {
+    classification: "backup_required",
+    reason:
+      "A 'running' receipt is the in-flight marker that keeps a second VACUUM/REINDEX window off a database already being maintained, so it must be reconciled after a crash. Kept backup_required rather than derived_rebuildable because no executable rebuild oracle reconstructs it.",
+  },
   presentation_screen_states: {
     classification: "backup_required",
     reason: "Presentation screen state is tied to live browser/screen surfaces.",
@@ -365,7 +370,12 @@ const POSTGRES_SQLITE_ONLY_TABLES = ["semantic_search_rowid"] as const;
 // pgvector/HNSW has no SQLite counterpart; the build-progress row only ever exists
 // under the Postgres backend, so it is the Postgres-side mirror of the
 // SQLite-only exception above rather than part of the shared storage seam.
-const SQLITE_POSTGRES_ONLY_TABLES = ["semantic_hnsw_index_build"] as const;
+// The derived-index maintenance receipt is Postgres-only for the same reason:
+// it records VACUUM/REINDEX windows, which SQLite's backend never runs.
+const SQLITE_POSTGRES_ONLY_TABLES = [
+  "postgres_derived_index_maintenance_receipts",
+  "semantic_hnsw_index_build",
+] as const;
 
 export function isInternalBackupCatalogTable(name: string): boolean {
   return SQLITE_INTERNAL_TABLES.has(name) || isShadowTable(name);

reference-implementation/server/deployment-diagnostics.ts

@@ -22,6 +22,7 @@
  */
 
 import { statfs } from "node:fs/promises";
+import type { PostgresDerivedIndexMaintenanceReceipt } from "./postgres-derived-index-maintenance.ts";
 import type { SemanticEmbeddingWarmStatus } from "./search-semantic.ts";
 
 // Shape of a connector manifest as far as diagnostics care. We do not depend
@@ -219,6 +220,7 @@ export interface DeploymentDiagnosticsInput {
   // explicit SQLite/failure `{ physical_bytes: null, top_relations: null }`
   // both surface as unmeasured.
   readonly physicalFootprint?: PhysicalFootprint | null;
+  readonly postgresDerivedIndexMaintenanceReceipt?: PostgresDerivedIndexMaintenanceReceipt | null;
   readonly runtimeCapabilities?: RuntimeCapabilityPosture | null;
 }
 
@@ -357,6 +359,8 @@ export interface DeploymentDiagnosticsReport {
     readonly provenance: "native" | "polyfill-registered";
     readonly semantic_stream_count: number;
   }>;
+  // Null means this process has not completed a Postgres maintenance run.
+  readonly postgres_derived_index_maintenance: PostgresDerivedIndexMaintenanceReceipt | null;
   readonly runtime_capabilities: {
     readonly bindings: {
       readonly browser: boolean;
@@ -415,6 +419,7 @@ export interface DeploymentDiagnosticsReport {
 const STATIC_ENV_ALLOWLIST: ReadonlyArray<{ readonly name: string; readonly secret?: boolean }> = [
   { name: "PDPP_STORAGE_BACKEND" },
   { name: "PDPP_DATABASE_URL", secret: true },
+  { name: "PDPP_POSTGRES_DERIVED_INDEX_MAINTENANCE_WINDOW" },
   { name: "AS_PORT" },
   { name: "RS_PORT" },
   { name: "AS_PUBLIC_URL" },
@@ -1058,6 +1063,10 @@ export interface DeploymentDiagnosticsRuntimeDeps {
   // it and degrades cleanly to unmeasured on absence or rejection — the page
   // never fails because the footprint could not be read.
   readonly getPhysicalFootprint?: () => PhysicalFootprint | Promise<PhysicalFootprint> | null;
+  readonly getPostgresDerivedIndexMaintenanceReceipt?: () =>
+    | PostgresDerivedIndexMaintenanceReceipt
+    | Promise<PostgresDerivedIndexMaintenanceReceipt>
+    | null;
   readonly getRuntimeCapabilityPosture?: () => RuntimeCapabilityPosture | Promise<RuntimeCapabilityPosture> | null;
   readonly listRegisteredConnectorIds: () => Promise<readonly string[]>;
 }
@@ -1113,6 +1122,9 @@ export async function collectDeploymentDiagnostics(
   // missing dep) degrades to unmeasured rather than failing the whole page;
   // the builder collapses null/undefined to `physical_bytes: null`.
   const physicalFootprint = await resolveOptionalDep(deps.getPhysicalFootprint);
+  const postgresDerivedIndexMaintenanceReceipt = await resolveOptionalDep(
+    deps.getPostgresDerivedIndexMaintenanceReceipt
+  );
   const diskHeadroom = await resolveOptionalDep(deps.getDiskHeadroom);
   const pgDiskHeadroom = await resolveOptionalDep(deps.getPgDiskHeadroom);
 
@@ -1129,6 +1141,7 @@ export async function collectDeploymentDiagnostics(
     manifests,
     pgDiskHeadroom,
     physicalFootprint,
+    postgresDerivedIndexMaintenanceReceipt,
     runtimeCapabilities,
   });
 }
@@ -1159,6 +1172,7 @@ export function buildDeploymentDiagnostics(input: DeploymentDiagnosticsInput): D
       },
     },
     manifests: summarizeManifests(input.manifests),
+    postgres_derived_index_maintenance: input.postgresDerivedIndexMaintenanceReceipt ?? null,
     runtime_capabilities: buildRuntimeCapabilityReport(input.runtimeCapabilities ?? null),
     semantic: {
       backend: {

reference-implementation/server/index.ts

@@ -230,6 +230,11 @@ import {
   schedulePostgresSemanticHnswMaintenance,
 } from "./postgres-storage.ts";
 import { createGenericProviderAuthDispatch } from "./provider-auth/generic-dispatch.ts";
+import {
+  getLastPostgresDerivedIndexMaintenanceReceipt,
+  parsePostgresDerivedIndexMaintenanceWindow,
+  runPostgresDerivedIndexMaintenance,
+} from "./postgres-derived-index-maintenance.ts";
 import { buildRecordVersionStatsEnvelope } from "./record-version-stats.ts";
 import {
   aggregateRecordsAcrossBindings,
@@ -975,6 +980,7 @@ const STARTUP_SUMMARY_EVIDENCE_MAX_RESUME_ROUNDS = 20;
 // running meaningfully more often than the durable state it sweeps
 // actually changes.
 const CONNECTOR_MAINTENANCE_SWEEP_INTERVAL_MS = 60_000;
+const POSTGRES_DERIVED_INDEX_MAINTENANCE_SWEEP_INTERVAL_MS = 15 * 60_000;
 const CONNECTOR_MAINTENANCE_EVIDENCE_SWEEP_MAX_DURATION_MS = 2000;
 const CONNECTOR_MAINTENANCE_EVIDENCE_SWEEP_PAGE_SIZE = 25;
 // Run-history backfill (terminal-read-architecture-fable-0730.md §9):
@@ -5213,6 +5219,7 @@ export function buildAsApp(opts: ServerOpts = {}) {
           getDiskHeadroom: () => probeDiskHeadroom(opts.dbPath || DB_PATH),
           getLexicalBackendPosture: () => getPostgresLexicalBackendState(),
           getLexicalBackfillProgress: () => getLexicalIndexBackfillProgress(),
+          getPostgresDerivedIndexMaintenanceReceipt: () => getLastPostgresDerivedIndexMaintenanceReceipt(),
           getPhysicalFootprint: () => collectPhysicalFootprint(),
           // biome-ignore lint/complexity/noExcessiveCognitiveComplexity: This protocol transition owns ordered state invariants that must remain local.
           getRuntimeCapabilityPosture: async () => {
@@ -8266,6 +8273,40 @@ export async function startServer(opts: ServerOpts = {}) {
   function stopConnectorMaintenanceSweep() {
     connectorMaintenanceSweepTimer.stop();
   }
+  // Database-reclaiming maintenance defaults to a UTC off-peak window. The
+  // operator can override or disable it; the runner owns the non-transactional
+  // VACUUM/REINDEX session and records each result for diagnostics.
+  const postgresDerivedIndexMaintenanceWindow = isPostgresStorageBackend()
+    ? parsePostgresDerivedIndexMaintenanceWindow()
+    : null;
+  let postgresDerivedIndexMaintenanceInFlight = false;
+  const postgresDerivedIndexMaintenanceTimer = postgresDerivedIndexMaintenanceWindow
+    ? createBrowserSurfaceLeaseSweepTimer({
+        intervalMs: POSTGRES_DERIVED_INDEX_MAINTENANCE_SWEEP_INTERVAL_MS,
+        onSweepError: (err: unknown) => {
+          logger.warn?.(
+            { err: err instanceof Error ? err.message : String(err) },
+            "postgres derived-index maintenance tick failed"
+          );
+        },
+        sweep: async () => {
+          if (postgresDerivedIndexMaintenanceInFlight) {
+            logger.info?.("postgres derived-index maintenance skipped because a prior run is still active");
+            return;
+          }
+          postgresDerivedIndexMaintenanceInFlight = true;
+          try {
+            const receipt = await runPostgresDerivedIndexMaintenance({ window: postgresDerivedIndexMaintenanceWindow });
+            logger.info?.({ receipt }, "postgres derived-index maintenance completed");
+          } finally {
+            postgresDerivedIndexMaintenanceInFlight = false;
+          }
+        },
+      })
+    : null;
+  function stopPostgresDerivedIndexMaintenance() {
+    postgresDerivedIndexMaintenanceTimer?.stop();
+  }
   let schedulerManager: {
     cancelRun: (runId: string) => { status: string; run_id: string };
     refresh: () => Promise<void>;
@@ -8814,6 +8855,8 @@ export async function startServer(opts: ServerOpts = {}) {
   // every deployment.
   connectorMaintenanceSweepTimer.stopWhenAllClosed([asServer, rsServer]);
   connectorMaintenanceSweepTimer.start();
+  postgresDerivedIndexMaintenanceTimer?.stopWhenAllClosed([asServer, rsServer]);
+  postgresDerivedIndexMaintenanceTimer?.start();
   const deliveryWorkerLeases =
     opts.startClientEventDeliveryWorker === false
       ? []
@@ -8868,6 +8911,7 @@ export async function startServer(opts: ServerOpts = {}) {
     // maintenance sweep timer (shell retirement, attention expiry, bounded
     // evidence-sweep round — see connector-maintenance-sweep.ts).
     stopConnectorMaintenanceSweep,
+    stopPostgresDerivedIndexMaintenance,
   };
 }
 
@@ -9899,6 +9943,7 @@ if (process.argv[1]?.endsWith("server/index.ts")) {
     stopBrowserSurfaceLeaseSweep: StartServerResult["stopBrowserSurfaceLeaseSweep"] | null;
     stopClientEventDeliveryWorker: StartServerResult["stopClientEventDeliveryWorker"] | null;
     stopConnectorMaintenanceSweep: StartServerResult["stopConnectorMaintenanceSweep"] | null;
+    stopPostgresDerivedIndexMaintenance: StartServerResult["stopPostgresDerivedIndexMaintenance"] | null;
   } = {
     abortStartupBackfill: null,
     asServer: null,
@@ -9911,6 +9956,7 @@ if (process.argv[1]?.endsWith("server/index.ts")) {
     stopBrowserSurfaceLeaseSweep: null,
     stopClientEventDeliveryWorker: null,
     stopConnectorMaintenanceSweep: null,
+    stopPostgresDerivedIndexMaintenance: null,
   };
   const exitOnSignal = (signal: string) => async () => {
     if (shuttingDown) {
@@ -9990,6 +10036,7 @@ if (process.argv[1]?.endsWith("server/index.ts")) {
     } catch {}
     server.stopBrowserSurfaceLeaseSweep?.();
     server.stopConnectorMaintenanceSweep?.();
+    server.stopPostgresDerivedIndexMaintenance?.();
     await server.stopClientEventDeliveryWorker?.();
     // In-flight connector runs are deliberately NOT drained here.
     //
@@ -10053,6 +10100,7 @@ if (process.argv[1]?.endsWith("server/index.ts")) {
       server.stopBrowserSurfaceLeaseSweep = result.stopBrowserSurfaceLeaseSweep;
       server.stopClientEventDeliveryWorker = result.stopClientEventDeliveryWorker;
       server.stopConnectorMaintenanceSweep = result.stopConnectorMaintenanceSweep;
+      server.stopPostgresDerivedIndexMaintenance = result.stopPostgresDerivedIndexMaintenance;
     })
     .catch((err) => {
       closePostgresStorage().finally(() => closeDb());