fix(helm): validate postgres dump before dropping the staging database

[yodem]

Follow-up to #3594, which is merged and running. The new drop-and-recreate sync failed four times today and emptied staging's Postgres in the process.

What happened

Dropping and recreating database sefariastaging...
DROP DATABASE
CREATE DATABASE
Restoring PostgreSQL data...
pg_restore: error: could not execute query: ERROR:  schema "public" already exists
Command was: CREATE SCHEMA public;

The dump carries an explicit CREATE SCHEMA public — visible in the pre-#3594 logs as TOC entry 3; 2615 2200 SCHEMA public postgres — because its source server owns public as a user schema. A database created TEMPLATE template0 already has that schema, so the entry collides on entry 3, and --exit-on-error --single-transaction rolls the whole restore back.

#3594 read those two --clean errors as artifacts of --clean. Only the DROP SCHEMA half was; the CREATE SCHEMA public half lives in the archive unconditionally and survived the switch to drop-and-recreate. --clean had tolerated it (errors ignored on restore: 2) and the table data landed anyway — which is exactly why this read as "the job fails but data is fine".

Impact

Staging is empty. Verified on postgres-18:

Check	Value
auth_user rows	0 (was 265,345)
sefariastaging size	9126 kB (production sefaria_auth: 547 MB)
indexes	70
django_migrations applied	all 38, at 2026-08-11 12:08:44

Every migration bears one timestamp three minutes after the 12:05:39 failure, and the only populated tables are Django's own bookkeeping (auth_permission, django_content_type, django_site). The app reconnected to the empty database and rebuilt the schema — the indexes are migration output, not restored data.

The fix

Inspect the archive with pg_restore --list before dropping anything:

• Refuse a dump with no post-data section. Production's dumps are still frozen at 40003385 bytes (identical 07-19 → 08-09), predating the --section=post-data fix, so a restore that did succeed would produce an unindexed database. This moves fix(helm): repair staging production-data sync #3594's post-restore index assertion to where it can still prevent harm; the post-restore check stays, now confirming post-data landed rather than discovering it was absent.
• Filter the schema entry via --use-list rather than pre-dropping public, so dumps that omit the entry (the PG15+ default) keep working.

Ordering matters beyond this one bug: backoffLimit: 1 gives two pods per job, and both ran the drop today. A retry re-dropped rather than retrying against surviving data. With the drop gated behind a validated dump, the retry is harmless.

Verification

• helm template renders; bash -n on the extracted container script passes.
• TOC greps exercised against three shapes: today's dump (refused, no drop), a post-data dump (passes, 1 schema entry filtered), and a PG15+ dump lacking the entry (passes, filter is a no-op).

Note for reviewers

This job authenticates to postgres-18 — the instance that also holds production's 547 MB sefaria_auth — as sefaria, which is a superuser. The exact-string guard on postgres.db != sefaria_auth is the only barrier, and it covers one name; pre-mdl (203 MB) and sefaria_audit (134 MB) sit on the same instance unguarded. Out of scope here, worth a follow-up.

The job will stay red until production ships #3594's backup fix and produces a dump with post-data — correctly so, and now without cost. Repopulating staging is a separate decision: the 08-09 dump would restore ~12-week-old unindexed data.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VaP5RZqJXTxVrhzfnToAdZ

[gitvelocity-reviewer]

📊 Code Quality Score: 8/100

Base Score 33 × ESF 0.25 = 8.25, rounded to 8

Category	Score	Factors
🔭 Scope	4/20	Single file (Helm CronJob YAML), single subsystem (ops/infra), no new APIs or services
🏗️ Architecture	3/20	No architectural change; adds a validation stage before an existing destructive operation; no new dependencies
⚙️ Implementation	8/20	Non-trivial shell scripting: pg_restore TOC extraction, grep-based TOC filtering, --use-list flag usage, handling PG14 vs PG15+ schema ownership differences
⚠️ Risk	9/20	Modifies a destructive database sync job in production infra; change reduces operational risk but domain is high-stakes; unchecked pg_restore --list exit code is a residual risk; no rollback plan documented
✅ Quality	7/15	Excellent inline comments explaining rationale for each check; no automated tests (appropriate for infra shell scripts); one unchecked exit code on pg_restore --list; empty restore.toc not guarded
🔒 Perf / Security	2/5	Defensive engineering against data loss (fail-fast before destructive DROP); no security surface changes

Was this score accurate? 👍 Yes · 👎 No

How this was scored →

_{Scored by GitVelocity · How are scores calculated?}

[yodem]

Verified end-to-end against the real production archive

Ran the real dump (postgres_09.08.26.dump, 40003385 bytes) through pg_restore 18.3 — the version this CronJob runs — in a throwaway pod with its own Postgres. No real database was touched.

Control (current master's logic, no --use-list):

pg_restore: error: could not execute query: ERROR:  schema "public" already exists
Command was: CREATE SCHEMA public;
  pg_restore exit=1
  tables left in ctl: 0

Reproduces the production failure exactly, empty database included.

With this PR's --use-list:

TOC: 126 entries, 124 to restore, 2 public-schema entries filtered.
  pg_restore exit=0
  tables restored: 23
  auth_user rows:  265345
  indexes:         0

265,345 rows — matching the count the runbook recorded for staging before the wipe. The 0 indexes is the post-data gap, and is exactly why the new pre-flight check refuses this dump rather than restoring it.

The archive carries two public-schema entries, not one

The first commit filtered only the first. The second commit widened the pattern:

3;    2615 2200 SCHEMA - public postgres
3167; 0 0       COMMENT - SCHEMA public postgres

The COMMENT applies only because this job connects as a superuser — a property of the current credentials, not of this code. It's now filtered too, and filter counts are logged, since a silently non-matching filter is the failure mode being guarded against.

Coverage

pg_restore 14 and 18 emit byte-identical --list output for this archive, so the patterns aren't tied to client version. The validation logic was extracted from the rendered chart and run against four TOC shapes:

Case	Verdict	Filtered
Real dump (pg_restore --list 14)	REFUSE — no drop	2
Real dump (pg_restore --list 18)	REFUSE — no drop	2
Real dump + post-data (post-backup-fix)	PROCEED	2
PG15+ dump, no schema entries	PROCEED	0 (no-op)

The third row is the one that matters going forward: once the backup fix ships a dump with post-data, this proceeds and restores.

Not exercised: a live drop/create/restore against staging itself. That can only happen once a post-data dump exists, and it will be this code's first production run.

[yodem]

Correction to my "Note for reviewers"

I claimed this job connects to the instance holding production's sefaria_auth. That's wrong, and it overstated the severity.

Production runs in a separate cluster (gke_production-deployment_us-east1-b). The staging sync runs in the dev cluster (gke_development-205018_us-east1-b) and reaches that cluster's postgres-18. The 547 MB sefaria_auth I measured is the dev cluster's, not production's. Production data was never reachable from this job.

The guard still matters, just for a different reason: the dev cluster's postgres-18 hosts the shared sefaria_auth alongside sefariastaging, plus pre-mdl (203 MB) and sefaria_audit (134 MB). The job's role (sefaria) is a superuser there, so a mis-set postgres.db would drop a database shared by every feature environment on that cluster. Bad, but not production.

Nothing in the code changes as a result — the guard and the pre-flight validation are unaffected. Correcting the record so the risk isn't misfiled.

Related, and the reason this PR can't go green yet

Production's backup CronJob still carries the hardcoded override #3594 removes:

- name: DATABASES_HOST
  value: postgres

The 2026-08-09 run logged Host: postgres — still dumping the pre-upgrade 10.3 instance. prod1-pgbouncer (DB_HOST: postgres) is pinned to it as well. Those are the only two workloads across both clusters that hardcode a Postgres host; everything else inherits from local-settings, which is why staging's restore already reads postgres-18.