Bucket backup: snapshot over the API, scrub secrets, and never trust a default

[lvwerra]

Why

The backup on this Space has never once succeeded. lvwerra/agent-manager-backup
holds a single commit — initial commit — after a day of hourly runs. This PR
replaces the step that fails, and closes a path by which it could have published
the bucket.

1. The mount had to go (measured)

The old step 2 handed a mounted folder to hf upload and let it enumerate. That
enumeration is the entire cost, and it is paid on every file whether wanted or not.

	files	time
stat-only walk of the FUSE mount (find /live -type f)	108,960	14m51s
the same tree over the Hub API (list_bucket_tree)	108,960	8 s
the same content on the Job's local disk	39,873	0.14 s

~8 ms per file on the mount; ~6,000× faster on local disk. The Job has 453 GB free,
so the download is bounded by bandwidth (~5 MB/s measured), not space.

The new pipeline: list over the API → decide the scope in code → copy just those
paths server-side by Xet hash into a private, ephemeral staging bucket (metadata
only, ~880 files/s) → download to local disk → scrub → verify → commit → delete
staging in a finally.

Measured end to end at prod scale: list 8.3 s → snapshot 12 s → download 129 s →
scrub 2 s → verify → commit 113 s. 4m24s, for a job that had never finished.

Being frozen matters as much as being fast. The live bucket is written while a run
works, and reading it directly died with not a file on the local file system when
the Space rotated a file away mid-upload. A snapshot cannot change underneath a run.

Three failure modes die with the mount, all observed here: the .cache/ path
rejection, Job timeout at 1h33m–2h07m (against the --timeout 3000s we ask for,
which is not honoured), and the vanishing file above.

2. Secrets never reach the history

A fourth failure mode can't be outrun: the Hub secret-scans dataset commits, and
buckets are not scanned. The scanner is TruffleHog, and it verifies a find by
authenticating with the token — which invalidates it. One of this Space's HF
tokens was invalidated exactly that way. So this isn't hygiene; a leaking commit
breaks the operator's credentials.

Two mechanisms, because either alone is insufficient:

• Skip files that exist to hold credentials, matched on the exact final path
  segment. A substring test for /credentials sails straight past
  /.credentials.json — that gap put a credentials file into a probe commit.
• Scrub stray occurrences, then verify, then commit. The verify is
  mandatory: a leftover match ends the run with no commit.

Patterns are boundary-anchored. A looser sk-[A-Za-z0-9_-]{24,} matched
sk-abstraction-and-chart-selection inside ordinary prose — a scrub that silently
rewrites real content is worse than one that over-reports. Re-scanning 73 flagged
files with tight patterns: 52 real, 21 false positives.

3. Nothing is private by default — and this file assumed otherwise

Both verified against the Hub:

call	result
create_bucket(id) with no private=	public
hf upload <new-dataset> (repo does not exist)	public

The second was live on main, under a comment asserting the opposite —
"Not existing yet is fine — the upload below creates it private" — with the
privacy gate skipped in exactly that case. On any Space whose backup dataset
didn't exist yet, the first run published the bucket. This Space escaped only
because its dataset was created by an earlier iteration and nothing has committed
since.

Now: every destination is created with visibility stated explicitly, then
read back and checked before a byte is written. The source bucket is gated too.
hf cp is fine — it refuses to auto-create in either form.

4. The default skip list, rebuilt on reproducibility

One criterion, and it is not size: a command can put it back. Excluding only
manager-owned, cache and temp directories drops 86,615 files / 8.05 GB of this
bucket and keeps 23,975 files / 3.78 GB.

An earlier draft of mine used a 10 MB per-file cap. It quietly discarded 1.81 GB
including four session transcripts of 292, 237, 142 and 141 MB — the most
history-shaped content on the bucket. No size rule survives anywhere, and
there's a test asserting that.

.git stays in: unpushed commits and staged changes live nowhere else, and a
partially-copied .git is a corrupt repo. Git worktrees stay in as a class, since
uncommitted work in a clone isn't reproducible from its remote. dist/build/
target stay opt-in — a source folder is called build often enough.

5. The mirror is gone

It was a second full copy that never deleted anything, so it archived every
credential the Space had ever held — 174 files present in the mirror and absent
from the live bucket. What it offered ("the Space exactly as it was, latest only")
is what the bucket already is; restarts never involved backup, since the bucket is
the persistent store.

The trade this accepts: a lost bucket now means re-logging in, because the history
holds no credentials by design. An existing mirror bucket is not deleted
automatically — that's the operator's call.

6. Restore defaults

The skip list is 30 names picked from measurements, so trimming it otherwise means
retyping them. The control compares against the local edit, not the server's
copy, so it disappears the moment it would be a no-op — verified in a browser: 2
chips → click → 30 chips → button gone.

Verified

• 33 tests pass, 10 new: tokens-not-globs and no -v in the launch, no
  AM_MIRROR anywhere, every destination created private and re-read, staging
  torn down in a finally, commit gated behind a self-verifying scrub, exact-name
  credential matching, empty-scope refusal, defaults-skip-only-reproducible,
  no-.git-and-no-size-rule, and restore-defaults semantics.
• tsc --noEmit and the web build are clean.
• The pipeline itself was run end to end against this Space's real bucket before
  being written up; the numbers above are from those runs, not estimates.

Not done

• The remaining credentials in the bucket are untouched. 50 shell_snapshots
  files with HF tokens, state/claude/.credentials.json, home/.config/gh/hosts.yml
  still hold live secrets in the bucket and its mirror. This PR keeps them out of
  the history; it does not clean or revoke them, which is the operator's call.
• --timeout being ignored is still unexplained. It matters less now that a run
  takes ~4 minutes, but a stuck run still occupies whatever the platform's limit is.
• Media is still included. workspaces/the-gatherer/work is 3,740 files / 1.37 GB
  of downloaded arxiv PDFs — re-downloadable in principle, but not by one command,
  so by the reproducibility rule it stays. It's about a third of the transfer if you
  want it gone.