Checkpoint metadata.json exceeds GitHub's 100 MB limit: files_touched is never deduplicated, and nested git worktrees are walked

[ecgang]

Summary

On a repo that uses git worktrees, entire/checkpoints/v1 grows a
metadata.json large enough that GitHub refuses the push outright. Once that
happens the checkpoint branch can never be pushed again without rewriting its
history, and the failure surfaces only as a warning attached to an unrelated
git push.

Two separate causes compound. Either one alone would be survivable.

Symptom

Every git push in the host repo now ends with:

[entire] Warning: failed to push entire/checkpoints/v1 after sync: push failed:
remote: error: File 53/596dc8fea0/1/metadata.json is 111.79 MB; this exceeds
GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected.
! refs/heads/entire/checkpoints/v1 -> [remote rejected] (pre-receive hook declined)

The user's own branches push fine, so this is easy to miss for a long time —
it reads as noise attached to a successful push.

Environment: Entire CLI 0.8.42, go1.26.4, darwin/arm64.

Cause 1 — files_touched is appended to, never deduplicated

That metadata.json is a single checkpoint record with
"checkpoints_count": 213. Measured on the blob:

file size	111.79 MB
lines	1,292,267
files_touched entries	1,292,257
unique entries	67,340
files_touched share of the file	105.6 MB of 111.79 MB (94%)

A 19× duplication factor. No single path dominates — the most repeated one
appears 77 times — so this is broad re-appending across checkpoints rather than
one pathological entry. Deduplicating alone takes the file to roughly 5.5 MB
and puts it back under the limit.

Cause 2 — nested git worktrees are walked as if they were project files

Of the 67,340 unique paths, broken down by top-level segment:

prefix	unique paths
.claude/ (agent worktrees live here)	43,184
worktrees/	21,833
the actual project	~1,241

96% of what is being recorded lives inside other git checkouts nested under
the repo root. Linked worktrees have their own .git and their own history;
their contents are not changes to the host repo. node_modules is correctly
excluded, so the walker already has an ignore mechanism — nested worktrees just
aren't in it.

This is the part that makes the growth unbounded in practice. Anyone driving
parallel coding agents accumulates worktrees, and each one multiplies the
recorded set.

Scale of the resulting branch

commits on entire/checkpoints/v1	8,842
files	32,799
unique blob bytes	1,336.9 MB
distinct blobs over 50 MB	3

Many identical 50 MB full.jsonl blobs exist, but git dedupes those by hash,
so they cost storage once. The 111.79 MB metadata.json is the only blob that
actually blocks the push.

Suggested fixes, in order of leverage

1. Deduplicate files_touched before writing. One line, ~19× smaller, no
   behaviour change — the array is a set in everything but type.
2. Do not descend into nested git worktrees. A directory containing a
   .git entry that is not the repo root is a separate checkout. This is where
   the remaining ~50× lives.
3. Store files_touched as a newline-delimited sidecar rather than inline
   in a JSON object. It streams, it chunks, and it stops one growing field from
   deciding whether the whole record is writable.
4. Check size before pushing and fail loudly. A blob over the remote's
   limit is unrecoverable once committed; catching it at write time is cheap,
   and catching it at push time is too late. The current warning is easy to
   scroll past.

Fix 1 is the emergency stop. Fix 2 is the actual bug. Fixes 3 and 4 are what
stop the next version of this.

Offer

Happy to send a PR for 1 and 2 — they look self-contained and the repo has CI
in test.yml. Say which shape you'd prefer for 2 (skip any dir containing
.git, versus consulting git worktree list), since that's a design call
rather than a mechanical one. I can also supply the checkpoint that triggered
this if a reproduction is useful, though it is 111 MB.

Disclosure: this report was researched and written with AI assistance
(Claude Code). Every number above is measured from the actual blob with
git cat-file, not estimated.

[MuskanPaliwal]

hi @ecgang , the measured numbers made this easy to chase down. i hit the same class of problem, so i took a shot at fixes 1 and 2 at this pr: #1937

a few things i found while tracing it.

the nested-worktree walking is really a go-git problem, not entire's logic. entire reads worktree status through go-git's Worktree.Status(). real git treats a directory containing a .git entry as a repository boundary and never descends into it. go-git skips entries named .git, but never asks whether a directory contains one, so it walks straight into every nested checkout and reports everything inside as untracked. my pr filters those entries out at entire's status choke point (gitrepo.Status), untracked entries only. tracked files are still reported wherever they sit, same as git. that fixes what gets recorded. the wasted walk itself is still go-git's to fix, and i plan to open an issue on go-git for that (likely a pr there too, since it looks like a single Lstat boundary check in their filesystem noder and every go-git consumer would benefit).

• on the shape question you asked for fix 2:
  i went with "skip any directory containing a .git entry" (file or directory, so linked worktrees and full nested clones both count) rather than consulting git worktree list. agent worktrees aren't always worktrees of the host repo. a foreign clone dropped under the repo root would slip past git worktree list entirely.

• on fix 1, one surprise:
  every merge path i could find already deduplicates, including on v0.8.42, so your duplicates enter through some path that bypasses the merge helpers. rather than only hunting that producer, the pr dedupes at the write boundary (WriteCommitted), so repeats can't reach the permanent record no matter where they come from. by your numbers that alone takes the blob from 111.79 MB to about 5.5 MB.

one caveat for your repo: the oversized blob is already committed, so the fix can't make your existing entire/checkpoints/v1 pushable again. that branch needs a one-time reset or rewrite of its history.

i kept 3 and 4 out of the pr to keep it reviewable, but both still seem worth doing.
question for the maintainers: @ashtom is there appetite for follow-ups? sketches of what i had in mind.

for 4 (size check): validate the serialized metadata.json size in the write path, since a blob over the remote's limit is unrecoverable the moment it's committed, and push time is too late. the hook can't block a user's git commit, so on breach i'd still write the record but replace files_touched with a count plus a truncation marker, and print a loud stderr warning. the record stays useful, the branch stays pushable, and the user actually finds out. limit-wise i'd default conservatively (50 MB, say) rather than hardcode github's 100, since other hosts draw the line elsewhere. happy to pick this one up if the shape sounds right.

for 3 (sidecar): a newline-delimited files_touched.txt next to metadata.json, with the metadata carrying a pointer and a count, and readers falling back to the inline array for existing checkpoints. transcripts already chunk (full.jsonl), so the sidecar could reuse that mechanism instead of inventing one. since this changes the format every metadata.json reader depends on, i'd want a maintainer's take before attempting it.

[ecgang]

Thanks for picking this up, and for going after the cause rather than the symptom — the go-git boundary diagnosis is the part I had wrong in the issue. I filed this as "entire walks nested worktrees"; you've shown it's go-git reporting them and entire believing it. That's a better framing and it moves the durable fix upstream.

I pulled #1937 and checked it against the numbers in the issue. What I verified locally, on your branch rebased on current main:

• Both new tests fail without the source change and pass with it. Reverting just the two non-test files, TestStatus_SkipsNestedCheckouts fails on vendor/lib/inner.go, vendor/lib/src/deep/leaf.go and worktrees/agent-1/task.md all present in the status map — which reproduces the reported bug directly — and TestWriteCommitted_DeduplicatesFilesTouched fails with [b.go a.go b.go a.go a.go].
• go test ./cmd/entire/cli/gitrepo/... ./cmd/entire/cli/checkpoint/... passes, and mise run lint reports 0 issues.

On fix 1 — the dedup reaches both live backends. Worth stating explicitly since the change is one line: writeSessionToSubdirectory is on *treeWriter, and gitRefsStore embeds *treeWriter (refs_store.go:47), so the git-branch and git-refs backends both inherit it. The root summary is covered transitively: reaggregateFromEntries (persistent.go:958) merges from the per-session metadata blobs, which are now deduped before they're written. So the root metadata.json was never the gap — the per-session file written raw from opts.FilesTouched was, and that's exactly what you fixed.

Two small things I'd suggest before it lands, neither of them a defect:

A gitRefsStore case in the dedup test. TestWriteCommitted_DeduplicatesFilesTouched only builds a NewGitStore. The shared *treeWriter means git-refs is structurally covered, and I traced it — but it's covered by argument rather than by test, and this bug reached a permanent record once already. Same fixture, a few lines.

mergeFilesTouched returns nil for empty input, which changes the JSON. api/checkpoint/metadata.go:353 and :506 both tag files_touched without omitempty, so an empty non-nil slice marshals as [] and a nil one as null. Before the change opts.FilesTouched passed through as-is; now an empty input comes back nil. It's reachable: committedFilesExcludingMetadata builds make([]string, 0, len(committedFiles)) and returns it non-nil when every committed file is filtered out as protected. Harmless to any reasonable reader, but it's a wire-format change riding along with a bug fix, so it's worth knowing about rather than discovering later.

On fsstore.go:321 — it still writes opts.FilesTouched raw. I'd flagged it as a live gap; it isn't. The file's own header says it is deliberately not registered by production code, and it's wired in only through registerForTesting. The reason to fix it anyway is that it's the worked example someone copies when they write the next real backend, and it currently models the pattern this PR exists to remove.

On the producer you couldn't find: I looked too, and came up empty in the same way. For the record, so nobody repeats the search — ExtractModifiedFiles dedupes through a fileSet in both the Claude Code and Droid extractors, filterFilesTouched preserves uniqueness on both branches (the fallback builds from a map, and the filtering branch only removes), RecordFilesTouched and the two SaveStep paths all go through mergeFilesTouched, and resolveFilesTouched copies an already-merged slice. So either the producer is on a path neither of us has looked at, or the record was written by a version older than the one we're both reading.

Which is the argument for where you put the fix. An invariant enforced at the write boundary doesn't require anyone to find the producer first, and it holds against whichever one it turns out to be. Given the failure mode here is unrecoverable once committed, I'd take that over a targeted fix even if we did find it.

On fix 2 — one deliberate-choice question. Real git doesn't report nothing for a nested checkout; it reports the boundary itself as a single entry:

$ git status --porcelain
?? own.txt
?? vendor/

The filter drops every entry under the boundary and adds nothing, so the directory disappears from status entirely. For files_touched that's clearly what you want — recording worktrees/agent-1/ as a touched file is no more correct than recording its 21,833 children. I mention it only because the comment on filterNestedCheckouts says it matches git, and on this one point it's deliberately quieter than git. Might be worth a sentence in the comment so the next reader doesn't "fix" it.

I also checked the consumers, since this changes what every caller of gitrepo.Status sees. There are five. The one worth caring about is rewind.go:1089, where a status that looks clean suppresses "you have uncommitted changes that will be LOST". That's safe: tracked entries survive the filter, and rewind restores from the host repo's tree, which can only contain tracked paths — so the filter cannot hide anything rewind would destroy. The other four feed file-change detection, where dropping foreign checkouts is the point.

On the walk. Agreed it's go-git's to fix, and worth knowing there's precedent in this repo for how much it costs: gitrepo/status.go carries a comment about gitignore.ReadPatterns not threading parent patterns into its recursive walk, which left a single Status() call at 5.25s until the e2e artifacts rule moved into e2e/.gitignore. Same class of problem, same walk. So the Lstat boundary check you're proposing upstream would help entire twice over. If you open that issue, link it here.

Worth adding for anyone reading this later: the .gitignore route is not a substitute. In the affected repo the 65k paths were under .claude/ and worktrees/, and entire can't ignore directories in a user's repo on the user's behalf. The status choke point is the only layer that can fix this for everyone.

---

On 3 and 4 — I'm the reporter here, not a maintainer, so treat this as a vote rather than a decision. @ashtom's call.

Fix 4 (size check) I'd like to see, and your shape sounds right to me. The reasoning that convinces me is the one you gave: over-limit is unrecoverable the moment it's committed, and pre-push is too late. That's what makes this different from an ordinary bloat problem — there's no recovery path that doesn't involve rewriting a branch.

Two things that might help it land. First, there's already a house number for this: compact transcript generation is skipped when the output exceeds a 50MB blob cap, and full.jsonl chunks rather than fails. So 50 MB isn't an arbitrary conservative pick, it's the cap this codebase already uses — which is a much easier argument than reasoning about GitHub's 100. Second, I'd check the size on the serialized record rather than on files_touched alone, so a future field that grows can't reintroduce the same failure through a different door.

The one cost worth naming: replacing files_touched with a count discards attribution data for that checkpoint, which is a real loss and not just a truncation. Worth pairing with 3 so the breach path has somewhere to put the list instead of dropping it.

Fix 3 (sidecar) I'd defer to a maintainer, as you suggested. Reusing full.jsonl's chunking rather than inventing a mechanism is clearly the right instinct. My only note is that pointer-plus-count with reader fallback means every reader now has two shapes to handle, forever, and existing checkpoints keep the inline array — so the fallback isn't transitional, it's permanent. That's fine, but it should be a decision rather than a side effect.

On the caveat about my branch — confirmed, and no action needed on your side. The oversized blob is already committed and this can't make it pushable. I'll reset that branch's history myself. Worth flagging that anyone else who hits this arrives with the same problem, so whatever lands for 4 should probably say what to do about a branch that's already over.

[MuskanPaliwal]

review feedback is addressed on #1937. still planning the go-git issue for the walk itself, will link it here once it is up. for 3 and 4, waiting on a maintainer's call. happy to start 4 with the refinements suggested above (house 50 MB cap, size check on the whole serialized record).

[MuskanPaliwal]

the go-git side turned out to already have an open issue: go-git/go-git#1896. i added findings there (confirmed on their current v6 main, and it covers all three nested-checkout forms, not just worktrees) plus an offer to send the boundary-check PR.