Skip to content

test(gate-concurrency): drop the archived receipts and replay gate - #90

Merged
tnunamak merged 1 commit into
mainfrom
test/gate-concurrency-evidence-trim
Sep 10, 2026
Merged

test(gate-concurrency): drop the archived receipts and replay gate#90
tnunamak merged 1 commit into
mainfrom
test/gate-concurrency-evidence-trim

Conversation

@tnunamak

Copy link
Copy Markdown
Contributor

Problem

reference-implementation/docs/receipts/gate-concurrency-20260903.tar.gz held the raw receipts and transcripts from one day of test runs: two runs of the same tree, one at file-concurrency cap 2 and one at cap 8. A test under reference-implementation/scripts/evidence/ re-derived each receipt's counts, failure names and digests from those archived bytes, and a path-triggered workflow (.github/workflows/gate-concurrency-evidence.yml) ran that replay whenever any of the five files changed.

That archive is a snapshot of a tree that has since moved on. Replaying it cannot tell you anything about the cap as it behaves today — it only confirms the archived bytes are unchanged, which is a property of the archive rather than of the concurrency cap. Meanwhile the archive carried real cost: 445 KB compressed, about 4 MB unpacked, two near-identical 1.9 MB transcripts, plus a workflow and a 371-line test whose only job was to re-check it. The durable result of that measurement is a handful of numbers, not four megabytes of transcript.

Change

The measurement's lasting content now lives as a table in reference-implementation/docs/gate-concurrency.md: both caps, elapsed times, selected-file and assertion counts, the pass/fail/skip split, the fact that both runs produced the same 396 failure identities, and the exit codes. The document keeps the honest reading of that result — failure-set equality for one pair on one host, not a green suite and not proof either cap is safe — and now states that ongoing CI runs at the current cap are the evidence going forward.

Removed:

  • reference-implementation/docs/receipts/gate-concurrency-20260903.tar.gz
  • reference-implementation/docs/receipts/gate-concurrency-20260903.summary.json
  • reference-implementation/scripts/evidence/gate-concurrency-receipts.test.ts (the directory is now empty and gone)
  • .github/workflows/gate-concurrency-evidence.yml

Kept: reference-implementation/scripts/gate-concurrency-receipt.ts and reference-implementation/scripts/gate-concurrency-receipt.test.ts. These are not archive-only. The runner's standalone discovery walks scripts/ as a selection root (COLOCATED_TEST_DIRS in reference-implementation/scripts/run-tests-discovery.ts), so that unit test already runs as part of the main gate — the deleted workflow was not what kept it alive. The replay test under scripts/evidence/ was the opposite case: that walk is non-recursive, so nothing selected it, which is exactly why it needed a dedicated workflow.

One documentation point was worth preserving on its own merits and has been rewritten to stand without the archive: completed_files is derived from the exit code (run-tests.ts writes failed ? 0 : results.length), so it reads 0 for any run that exits non-zero and is not an observed per-file completion count.

How to verify

Confirm the kept test still passes:

cd reference-implementation
node --test --experimental-strip-types scripts/gate-concurrency-receipt.test.ts

Expect 10 passing, 0 failing.

Confirm test discovery is intact — that the kept test is still selected and that nothing points at a deleted file:

cd reference-implementation
node --experimental-strip-types -e '
const root = process.cwd(); const fs = require("node:fs");
import(root + "/scripts/run-tests-discovery.ts").then(async (m) => {
  const rel = await m.discoverTestFiles(root, root + "/test");
  console.log("discovered:", rel.length);
  console.log("kept test selected:", rel.some(f => f.includes("gate-concurrency-receipt.test.ts")));
  console.log("selected-but-missing:", rel.filter(f => !fs.existsSync(root + "/" + f)));
});'

Expect 1034 discovered, the kept test selected, and an empty missing list.

Confirm no reference to a removed artifact survives anywhere:

grep -rn "gate-concurrency-20260903\|scripts/evidence\|gate-concurrency-evidence" . \
  | grep -v node_modules | grep -v "\.git/"

Expect no output.

This commit is scoped test(gate-concurrency) so it cuts no package release: .releaserc.yaml releases only on feat/fix/perf/revert (or a breaking marker) and only for the connector-protocol, collector-runtime or local-collector scopes, so the type and the scope each independently rule out a publish.

Assisted-by: AI

The archive was a snapshot of one day's tree, so it went stale as soon
as the tree moved, and replaying it only ever proved the archive was
unchanged. Fold the pair's durable result -- the two caps, timings,
counts, equal failure sets and exit codes -- into a table in
docs/gate-concurrency.md, and state that ongoing CI runs at the current
cap are the evidence from here on.

Removes the tarball, its summary, the scripts/evidence/ replay test and
the path-triggered workflow that re-verified them. Keeps
scripts/gate-concurrency-receipt.ts and its unit test: the runner's
standalone discovery walks scripts/ (run-tests-discovery.ts
COLOCATED_TEST_DIRS), so that test runs in the main gate rather than
only under the deleted workflow.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Sep 10, 2026
The policy module cites this document as the reasoning behind its cap, but the
document described the cap it replaced. It said file concurrency was 2 by
default, that 8 was an override and "not the effective default on any host",
and it advised against changing the default at all. A reader following the
citation to check the 8 was told the 8 does not exist.

It also named four tests that failed only at high concurrency, called them
contention artifacts rather than code defects, and used them as the reason to
stay slow. That reading was wrong, and this branch is where it was disproved:
the three SQLite writer-path tests were closing the database while deferred
index maintenance they had started was still running, and the upload test let a
detached validation task outlive it. All four were real defects and are
repaired at the root here. The document now says so, and no longer lists the
tests by title, since the titles were only ever a symptom list.

Rewritten to state the current policy: caps of 8 for the memory profile and 2
for PostgreSQL, clamped by CPU count and selected file count, floored at 1,
with a positive override left unclamped. The wall-clock pair behind the number
is kept along with its own limits, including that both runs failed identically
and so establish nothing about safety on their own.

Structured to match the rewrite in #90, which trims the archive references from
the same file: shared headings, and the sections that PR owns are left
byte-identical, so the rebase touches only the lines changed here.

Documentation only.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
@tnunamak
tnunamak merged commit c421c40 into main Sep 10, 2026
14 checks passed
@tnunamak
tnunamak deleted the test/gate-concurrency-evidence-trim branch September 10, 2026 04:01
tnunamak added a commit that referenced this pull request Sep 10, 2026
The policy module cites this document as the reasoning behind its cap, but the
document described the cap it replaced. It said file concurrency was 2 by
default, that 8 was an override and "not the effective default on any host",
and it advised against changing the default at all. A reader following the
citation to check the 8 was told the 8 does not exist.

It also named four tests that failed only at high concurrency, called them
contention artifacts rather than code defects, and used them as the reason to
stay slow. That reading was wrong, and this branch is where it was disproved:
the three SQLite writer-path tests were closing the database while deferred
index maintenance they had started was still running, and the upload test let a
detached validation task outlive it. All four were real defects and are
repaired at the root here. The document now says so, and no longer lists the
tests by title, since the titles were only ever a symptom list.

Rewritten to state the current policy: caps of 8 for the memory profile and 2
for PostgreSQL, clamped by CPU count and selected file count, floored at 1,
with a positive override left unclamped. The wall-clock pair behind the number
is kept along with its own limits, including that both runs failed identically
and so establish nothing about safety on their own.

Structured to match the rewrite in #90, which trims the archive references from
the same file: shared headings, and the sections that PR owns are left
byte-identical, so the rebase touches only the lines changed here.

Documentation only.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant