Repository navigation
ci: every check is a local entry point; workflows call make check-<suite> - #2962
Open
colinmegill wants to merge 9 commits into
Open
colinmegill wants to merge 9 commits into
colinmegill wants to merge 9 commits into
Conversation
…ite> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… DynamoDB port With CHECK_PORT_BASE set, the simulator's own healthcheck (port 3000 inside the container) failed, and the E2E server container still looked for DynamoDB on host port 8000. Defaults (CI) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…fault Readiness waits and probes that did not fail the hosted jobs before (the fixed sleeps and the health probes ending in '|| true') warn and continue again; starting containers stays fatal. The Compose health wait is bounded by the old sleep (30 s; 20 s for Delphi). The E2E probe is back on /api/v3/participationInit. CHECK_STRICT_READY=1 makes readiness fatal. A suite whose workflow pins a runtime (Node, Python, JDK and Clojure CLI, the crate's Rust toolchain) now stops with a one-line message when that version is not available; CHECK_ALLOW_RUNTIME_MISMATCH=1 warns and continues. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ordings entry point The selector gains the inputs it missed (the two local requirements files, docker-compose.yml and the copied deploy hooks for Delphi, test.env for the unit tests, and others). ci/local/test_changed_suites.py reads each suite script, extracts the repository paths it names and fails when the selector would not run that suite for one of them; it runs in make check-fast. Routing recordings get ci/local/routing-recordings.sh and the workflow calls make check-routing-recordings; its name, triggers, path filters, job, service container and cache are unchanged. Comments say 'fast loop' throughout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ption Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Contributor
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes: none (CI and tooling only; the same checks, the same required names, now runnable locally with one command)
Parity: the same entry point on a build Mac (arm64, colima) and on hosted CI (x86_64)
Box: Mac mini M4 Pro, colima VM (aarch64), this branch's tree committed locally on the box (tree
eb26b765, the pushed head6fb223acc; re-run in full after the review fixes below). Hosted: the latest completed run of each check on edge (or on a PR to edge for PR-only checks).make check-…)CHECK_STRICT_READY=1); one run 1,651 pass, 1 skip, 1 fail (conversation-stats "until", skew 8.7 ms). Earlier trees: the same single failure at 32 msrustproto-0)math requires JDK 16.0.2 and Clojure CLI 1.10.1.693; found JDK none, Clojure CLI none; install the hosted version or set CHECK_ALLOW_RUNTIME_MISMATCH=1. With the override: 76 tests, 5,242 assertions, 0 failures (temurin-21 container). With the exact hosted pair (the entry point run inside a JDK 16.0.2 container with CLI 1.10.1.693 installed): 76 tests, 5,242 assertions, 0 failuresmake check-fast)Wall clock on the box (final tree,
CHECK_PORT_BASE=56100, box otherwise idle):make check-fast38 s on a fresh checkout (npm installs included), 23 s warm;make check34 min 36 s (2,076 s). In that run every suite passed except three: the coordinator's known red (815 s), the clock-skew test in server integration (91 s; it passed in two later runs), and math, which stopped at once because the box has none of the pinned JDK and Clojure CLI (see the table). Largest: coordinator 815 s, Delphi 427 s, E2E 357 s, collective statement 135 s, server integration 91 s, routing recordings 88 s. Every pinned runtime other than math's was satisfied on the box (Node 24.21.0, Node 22.23.1 for the collective-statement recordings, Python 3.12.15, Rust 1.98.1). No readiness warning was printed in any run. Hosted push-to-finished today is about 23 min (p50) with the jobs running in parallel.Review fixes (commits
aa588e935,d720d98ce,6fb223acc)An independent review found that the first version could change a hosted verdict. These commits restore the old verdicts and close the gaps it listed:
|| true. The first version of this branch made those waits fatal. Now starting a container is fatal (as it was), and every readiness wait or probe (Compose health, the init containers, OIDC, DynamoDB, SES, the E2E server probe) prints aWARNING:line and the run continues (as it did). The Compose health wait is bounded by the old sleep and ends early when everything is healthy; if it fails early, the rest of the old sleep is waited out. The E2E probe is back on/api/v3/participationInit.CHECK_STRICT_READY=1makes readiness fatal (and waits up to 300 s); it is off by default.make check-changedselects every suite whose inputs change. Added the inputs it missed (the twoci/local/requirements-*.txtfiles,docker-compose.ymland the three deploy hook scripts and other copied inputs for Delphi,ci/local/clock-skew.cjs, and others).ci/local/test_changed_suites.pyreads each suite script, extracts the repository paths it names (directories, files, top-level files) and fails if the selector would not run that suite for one of them. It was checked by mutation: removing a mapping, or adding an unmapped input to a script, fails it. It runs inmake check-fast. Its limit: an input read only by a program the script calls is not seen.ci/local/routing-recordings.sh,make check-routing-recordings, and the workflow calls it. The workflow's name, triggers, path filters, job name, service container and npm cache are unchanged.rust-toolchain.toml), the entry point stops with one line when that version is not available.CHECK_ALLOW_RUNTIME_MISMATCH=1warns and continues.docs/local-checks.mdlists the versions per suite. On hosted runners the workflows' own setup steps satisfy every pin.docs/local-checks.mdstates that the coordinator workflow still runs its steps inline, why, and thatmake check-coordinatoris the local equivalent.What changed
ci/local/<suite>.sh: one script per check, holding everything that used to be inline workflow YAML: certificates (ci/local/certs.sh: mkcert from PATH or a pinned, checksum-verified v1.4.4 for the host's OS and CPU; the CA stays in the checkout under.simulacrum/caand is never installed into a trust store), an edited copy oftest.env(the tracked file is never edited), env lists, the Delphi copy-in steps, service start withdocker compose up --waitplus readiness polls that end early instead of fixed sleeps (non-fatal by default, as the old probes were; see the review fixes), logs on failure, and teardown.make check-<suite>for each;make check(every check CI runs, CI's order, a summary table, all suites even after a failure);make check-fast(seconds, no containers);make check-changed(suites whose inputs differ fromBASE_REF, defaultorigin/edge; path-filtered workflows use their filters plus every input the entry point reads);make check-ungated(legacy client tests, cdk tests, theci/Python tests: entry points, not gates).CHECK_PROJECT(defaultcheck-<suite>-<hash of the checkout path>) andCHECK_PORT_BASE(every published host port moves to base + offset; defaults are CI's ports). Runs wait for their host ports to be free instead of colliding.docker-compose.test.yml: published host ports and the delphi image name are variables whose defaults are today's values; the server'senv_filehonoursSERVER_ENV_FILE(asdocker-compose.ymlalready does). With no variables set,docker compose configis unchanged.run: make check-<suite>. Triggers, path filters, concurrency groups, caches, artifacts, the coverage comment, job names and required-check names are unchanged. The twoservices:containers (Delphi characterization, queue-rs, collective statement) are started by their entry points instead, on the same images, digests and ports.BASE_REF(CI passesorigin/<base branch>).docs/local-checks.md(for self-hosters and automation), a pointer from the README and CLAUDE.md.Harness assumptions found, and what the entry point does instead
mkcert -install): Node reads it viaNODE_EXTRA_CA_CERTSand the containers mount it, as before.test.envwas edited in place and copied to.env,server/.envande2e/.env; the entry points write an edited copy under.check/and export the same values (the readers use dotenv without override, so this is equivalent). A developer's own.envfiles are left alone.sleep 20/sleep 30and|| truehealth checks: replaced byup --waitbounded by the old sleep, a bounded wait for the one-shot init containers, and polls of the OIDC JWKS URL,/api/v3/participationInitand the published ports. None of them fails the job by default, as before.psql; it now uses the postgres image's ownpsqlon the stack network (same script, same files).npm install dotenvbeforenpm ci(which rewrites package.json on a dev machine);npm ciinstalls dotenv from the lockfile.$(mkcert -CAROOT); the entry points exportCAROOTand put the pinned mkcert on PATH.coordinator-tls-*, and its fixture's temp dir must be visible to Docker (colima mounts only the work tree): the TLS project iscoordinator-tls-<project>andTMPDIRis inside the checkout. The campaign output must be new and outside the checkout: the default is../.check-runs/<checkout>/coordinator-<time>. The campaign judges committed source only (CHECK_COORDINATOR_LOCAL=1passes its--allow-local-changesreview mode).http://localhostandhttps://localhost:3000, and the collective-statement harness refuses stores off ports 5481/8481: those suites keep their ports whenCHECK_PORT_BASEis set and wait for them to be free.Findings (not fixed here; questions)
Needs a ruling
coordinator-ci.ymlis left untouched. Its own control test,coordinator-rs/ci/tests/test_required_campaign.py::test_workflow_cannot_filter_out_the_required_job_or_skip_failure, requires the literal stringsnpm ci --prefix serverandpython coordinator-rs/ci/run.pyin that workflow file (andrun.pyhashes the workflow as campaign source). Changing what that test asserts is out of scope here, so this is the one job that does not call its entry point;make check-coordinatorruns the same steps. Should the control acceptmake check-coordinatorinstead?The clock-skew test.
server/__tests__/integration/conversation-stats.test.ts"should accept until parameter" creates rows (timestamped by Postgresnow()) and immediately asks foruntil=Date.now()from the test process. On a Mac, Docker's VM clock runs ahead of the host: measured 32 ms on the box before a power cycle (the test failed in three runs), 9 ms after it (the test passed), and on the final tree 8.7 ms (failed once) and 4.5 ms (passed twice) (ci/local/clock-skew.cjs, whichcheck-server-integrationnow prints), so one row is "in the future" (expected 2, received 1). Hosted runners share one clock and pass. The test assumes one clock for the database and the test process.The OIDC simulator exits in the hosted Delphi job. That job never creates
.simulacrum/certs, so the simulator (which mounts~/.simulacrum/certsby default) exits 1 at start on hosted runners too; the fixedsleep 20hid it. It does not affect the Delphi tests.Making readiness failures fatal (available now as
CHECK_STRICT_READY=1, off by default). The hosted jobs have never failed on an unready service; they fail later, in the tests. Should the hosted jobs set it?The hosted
routingcheck is red on this pull request, and it is the guard working as written.server/characterization/routing/guard.cjsrefuses a change to the routing rules (which include.github/workflows/routing-recordings.yml) together with a change underserver/src/, and compares againstgithub.event.pull_request.base.sha. That value is the base commit recorded when the pull request was opened (669f217); it does not follow edge. The job checks out the merge with today's edge (80193c6), so server: comment moderation no longer sends commenter IP addresses to a third-party geolocation service #2961'sserver/srcfiles count as this pull request's changes, next to its one-step change to the routing workflow. The entry point passes the guard exactly the value the workflow passed before, so the workflow as it is on edge gives the same red for any pull request that touches the routing rules once edge gains a server commit. Locally, against the true base, the guard passes. Two ways out, both the owner's call: re-target the pull request's base (which refreshes the recorded commit), or have the routing workflow passorigin/<base branch>as the other two guards do (a change to that check's behaviour, so not made here).Also noted
delphi/tests/coordinator/test_step2_review_controls.py::test_actual_pca2_route_serves_generation_zero('"rustproto-0"' == '"0"'), reproduces on the box. Not touched.make check-mathnow requires that pair unless the override is set. Clojure dependency resolution is the only network access at test time.make check-client-alphaleavesclient-participation-alpha/.astro/types.d.tsmodified in the working tree (the Astro build rewrites it). Not touched.make check-fastonly; no workflow runs it.make check-fastrun on the freshly rebooted, loaded box lost a jest worker to SIGSEGV (a Node crash, no assertion); the next run passed. Recorded, not chased.Verification
make checkandmake check-fast, on this branch's exact tree (committed locally on the box: the coordinator refuses uncommitted source).docker compose -f docker-compose.test.yml --env-file test.env configwith no variables set gives the same ports, image names and environment as before.ci/local/, the Makefile,docker-compose.test.yml(variables with today's defaults), the workflows,.gitignore, and docs.bash -non all 28 scripts; the selector test;make check-fast;make check; math three ways (default stop, override, exact hosted pair); server integration again with and withoutCHECK_STRICT_READY=1; all on the pushed tree.🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s