Skip to content

ci: every check is a local entry point; workflows call make check-<suite> - #2962

Open
colinmegill wants to merge 9 commits into
edgefrom
ci/local-parity
Open

colinmegill wants to merge 9 commits into
edgefrom
ci/local-parity

Conversation

@colinmegill

@colinmegill colinmegill commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Observable changes: none (CI and tooling only; the same checks, the same required names, now runnable locally with one command)

Parity: the same entry point on a build Mac (arm64, colima) and on hosted CI (x86_64)

Box: Mac mini M4 Pro, colima VM (aarch64), this branch's tree committed locally on the box (tree eb26b765, the pushed head 6fb223acc; re-run in full after the review fixes below). Hosted: the latest completed run of each check on edge (or on a PR to edge for PR-only checks).

Suite (make check-…) Box: Mac mini M4 Pro, colima arm64 Hosted x86_64 (latest run on edge, or a PR to edge) Agree
server-integration 109 suites; 1,653 tests. On the final tree: two runs 1,652 pass, 1 skip (measured clock skew 4.5 and 4.6 ms; one of them with CHECK_STRICT_READY=1); one run 1,651 pass, 1 skip, 1 fail (conversation-stats "until", skew 8.7 ms). Earlier trees: the same single failure at 32 ms 109 suites; 1,652 pass, 1 skip (run 37247554387) yes when the test passes; no when the clock-skew test fails (see below)
contracts pass pass yes
vote-path-guard ok ok (PR run 37241521960) yes
e2e 34 specs; 213 tests: 212 pass, 1 pending 34 specs; 213: 212 pass, 1 pending (37247554347) yes
coordinator TLS: 7 cargo tests, 9 transport checks; controls 181 pass; pytest 290 pass, 1 fail (rustproto-0) TLS 7 + 9 pass; pytest exit 1, same assertion (37247554427) yes (both red, same test)
vote-sign-lint gate unit tests 32 OK; lint 0 new same (37247554349) yes
vote-gate fixture tests 24 OK; node 19/19; +1 pins match; v0 green; v1 exit 3 (every red expected); ratchet green same (37247554349) yes
delphi-characterization codec 8 Python + 8 jest; 11 fixtures regenerate; 272 cases byte for byte same (37247554276) yes
collective-statement harness rules 13/13; 51 cases, 26 provider requests, 0 refused; every listed mutation killed same (37247554395) yes
collective-statement-guard ok (PR-only) n/a
routing-recordings mechanism tests pass; replay: 104 routing cases pass; mutations pass (Node 22.23.3, throwaway PostgreSQL 17.11 on port base + 16) new on edge with #2960; same steps yes
lint 4 packages clean (e2e: 1 warning) same (PR run 37241521991) yes
delphi-python 3,894 pass, 234 skip, 44 xfail, 2 xpass (final tree); 3,842 / 234 / 44 / 2 at d634dca 3,894 / 234 / 44 / 2 at 9091212 (37247424825); 3,842 / 234 / 44 / 2 at d634dca (37240289001) yes
queue-rs clippy clean; 35 unit, 4 protocol, 19 integration same (PR run 37241521977) yes
client-alpha / client-admin / client-report format ok, 149 tests, build ok / 321 / 92 same yes
math The box has no JDK or Clojure CLI, so by default the suite now stops: math requires JDK 16.0.2 and Clojure CLI 1.10.1.693; found JDK none, Clojure CLI none; install the hosted version or set CHECK_ALLOW_RUNTIME_MISMATCH=1. With the override: 76 tests, 5,242 assertions, 0 failures (temurin-21 container). With the exact hosted pair (the entry point run inside a JDK 16.0.2 container with CLI 1.10.1.693 installed): 76 tests, 5,242 assertions, 0 failures same on JDK 16.0.2 yes
server-unit (part of server-integration) 52 suites, 907 tests n/a n/a
selector-test (in make check-fast) 94 suite/input pairs pass not run by any workflow n/a
ungated: client-participation / cdk / ci-python 157 pass / 145 pass / 742 pass, 5 skip not run by any workflow n/a

Wall clock on the box (final tree, CHECK_PORT_BASE=56100, box otherwise idle): make check-fast 38 s on a fresh checkout (npm installs included), 23 s warm; make check 34 min 36 s (2,076 s). In that run every suite passed except three: the coordinator's known red (815 s), the clock-skew test in server integration (91 s; it passed in two later runs), and math, which stopped at once because the box has none of the pinned JDK and Clojure CLI (see the table). Largest: coordinator 815 s, Delphi 427 s, E2E 357 s, collective statement 135 s, server integration 91 s, routing recordings 88 s. Every pinned runtime other than math's was satisfied on the box (Node 24.21.0, Node 22.23.1 for the collective-statement recordings, Python 3.12.15, Rust 1.98.1). No readiness warning was printed in any run. Hosted push-to-finished today is about 23 min (p50) with the jobs running in parallel.

Review fixes (commits aa588e935, d720d98ce, 6fb223acc)

An independent review found that the first version could change a hosted verdict. These commits restore the old verdicts and close the gaps it listed:

  • Readiness gives the same verdict as before. On edge the hosted jobs started the stack, slept a fixed time (30 s; 20 s for Delphi) and ran health probes ending in || true. The first version of this branch made those waits fatal. Now starting a container is fatal (as it was), and every readiness wait or probe (Compose health, the init containers, OIDC, DynamoDB, SES, the E2E server probe) prints a WARNING: line and the run continues (as it did). The Compose health wait is bounded by the old sleep and ends early when everything is healthy; if it fails early, the rest of the old sleep is waited out. The E2E probe is back on /api/v3/participationInit. CHECK_STRICT_READY=1 makes readiness fatal (and waits up to 300 s); it is off by default.
  • make check-changed selects every suite whose inputs change. Added the inputs it missed (the two ci/local/requirements-*.txt files, docker-compose.yml and the three deploy hook scripts and other copied inputs for Delphi, ci/local/clock-skew.cjs, and others). ci/local/test_changed_suites.py reads each suite script, extracts the repository paths it names (directories, files, top-level files) and fails if the selector would not run that suite for one of them. It was checked by mutation: removing a mapping, or adding an unmapped input to a script, fails it. It runs in make check-fast. Its limit: an input read only by a program the script calls is not seen.
  • Routing recordings (server: recorded behaviour and cost of next-statement selection #2960) has its entry point: ci/local/routing-recordings.sh, make check-routing-recordings, and the workflow calls it. The workflow's name, triggers, path filters, job name, service container and npm cache are unchanged.
  • Pinned runtimes fail by default. Where a workflow pins a runtime (Node 24, 22 or 22.23.1; Python 3.12; JDK 16.0.2 with Clojure CLI 1.10.1.693; the Rust channel in the crate's rust-toolchain.toml), the entry point stops with one line when that version is not available. CHECK_ALLOW_RUNTIME_MISMATCH=1 warns and continues. docs/local-checks.md lists the versions per suite. On hosted runners the workflows' own setup steps satisfy every pin.
  • Wording in the added files: "fast loop" and "automation". docs/local-checks.md states that the coordinator workflow still runs its steps inline, why, and that make check-coordinator is the local equivalent.

What changed

  • ci/local/<suite>.sh: one script per check, holding everything that used to be inline workflow YAML: certificates (ci/local/certs.sh: mkcert from PATH or a pinned, checksum-verified v1.4.4 for the host's OS and CPU; the CA stays in the checkout under .simulacrum/ca and is never installed into a trust store), an edited copy of test.env (the tracked file is never edited), env lists, the Delphi copy-in steps, service start with docker compose up --wait plus readiness polls that end early instead of fixed sleeps (non-fatal by default, as the old probes were; see the review fixes), logs on failure, and teardown.
  • make check-<suite> for each; make check (every check CI runs, CI's order, a summary table, all suites even after a failure); make check-fast (seconds, no containers); make check-changed (suites whose inputs differ from BASE_REF, default origin/edge; path-filtered workflows use their filters plus every input the entry point reads); make check-ungated (legacy client tests, cdk tests, the ci/ Python tests: entry points, not gates).
  • Several stacks per host: CHECK_PROJECT (default check-<suite>-<hash of the checkout path>) and CHECK_PORT_BASE (every published host port moves to base + offset; defaults are CI's ports). Runs wait for their host ports to be free instead of colliding.
  • docker-compose.test.yml: published host ports and the delphi image name are variables whose defaults are today's values; the server's env_file honours SERVER_ENV_FILE (as docker-compose.yml already does). With no variables set, docker compose config is unchanged.
  • Workflows: each job's inline steps become run: make check-<suite>. Triggers, path filters, concurrency groups, caches, artifacts, the coverage comment, job names and required-check names are unchanged. The two services: containers (Delphi characterization, queue-rs, collective statement) are started by their entry points instead, on the same images, digests and ports.
  • No guard needs GitHub context: the two golden guards take BASE_REF (CI passes origin/<base branch>).
  • docs/local-checks.md (for self-hosters and automation), a pointer from the README and CLAUDE.md.

Harness assumptions found, and what the entry point does instead

  • The mkcert download was fixed to linux/amd64; it is now a pinned v1.4.4 for the host's OS and CPU, checked against its SHA-256. The CA is no longer installed into the runner's trust store (mkcert -install): Node reads it via NODE_EXTRA_CA_CERTS and the containers mount it, as before.
  • test.env was edited in place and copied to .env, server/.env and e2e/.env; the entry points write an edited copy under .check/ and export the same values (the readers use dotenv without override, so this is equivalent). A developer's own .env files are left alone.
  • Fixed sleep 20/sleep 30 and || true health checks: replaced by up --wait bounded by the old sleep, a bounded wait for the one-shot init containers, and polls of the OIDC JWKS URL, /api/v3/participationInit and the published ports. None of them fails the job by default, as before.
  • E2E re-applied the migrations with the runner's apt-installed psql; it now uses the postgres image's own psql on the stack network (same script, same files).
  • E2E's auth step ran npm install dotenv before npm ci (which rewrites package.json on a dev machine); npm ci installs dotenv from the lockfile.
  • E2E and the server tests read the CA through $(mkcert -CAROOT); the entry points export CAROOT and put the pinned mkcert on PATH.
  • The coordinator's TLS test accepts only a project named coordinator-tls-*, and its fixture's temp dir must be visible to Docker (colima mounts only the work tree): the TLS project is coordinator-tls-<project> and TMPDIR is inside the checkout. The campaign output must be new and outside the checkout: the default is ../.check-runs/<checkout>/coordinator-<time>. The campaign judges committed source only (CHECK_COORDINATOR_LOCAL=1 passes its --allow-local-changes review mode).
  • The E2E specs assert http://localhost and https://localhost:3000, and the collective-statement harness refuses stores off ports 5481/8481: those suites keep their ports when CHECK_PORT_BASE is set and wait for them to be free.
  • The Delphi job makes no certificates, so the OIDC simulator exits there (on hosted runners too); the entry point starts every service as CI does and waits only for the ones the Delphi tests use.

Findings (not fixed here; questions)

Needs a ruling

  1. coordinator-ci.yml is left untouched. Its own control test, coordinator-rs/ci/tests/test_required_campaign.py::test_workflow_cannot_filter_out_the_required_job_or_skip_failure, requires the literal strings npm ci --prefix server and python coordinator-rs/ci/run.py in that workflow file (and run.py hashes the workflow as campaign source). Changing what that test asserts is out of scope here, so this is the one job that does not call its entry point; make check-coordinator runs the same steps. Should the control accept make check-coordinator instead?

  2. The clock-skew test. server/__tests__/integration/conversation-stats.test.ts "should accept until parameter" creates rows (timestamped by Postgres now()) and immediately asks for until=Date.now() from the test process. On a Mac, Docker's VM clock runs ahead of the host: measured 32 ms on the box before a power cycle (the test failed in three runs), 9 ms after it (the test passed), and on the final tree 8.7 ms (failed once) and 4.5 ms (passed twice) (ci/local/clock-skew.cjs, which check-server-integration now prints), so one row is "in the future" (expected 2, received 1). Hosted runners share one clock and pass. The test assumes one clock for the database and the test process.

  3. The OIDC simulator exits in the hosted Delphi job. That job never creates .simulacrum/certs, so the simulator (which mounts ~/.simulacrum/certs by default) exits 1 at start on hosted runners too; the fixed sleep 20 hid it. It does not affect the Delphi tests.

  4. Making readiness failures fatal (available now as CHECK_STRICT_READY=1, off by default). The hosted jobs have never failed on an unready service; they fail later, in the tests. Should the hosted jobs set it?

  5. The hosted routing check is red on this pull request, and it is the guard working as written. server/characterization/routing/guard.cjs refuses a change to the routing rules (which include .github/workflows/routing-recordings.yml) together with a change under server/src/, and compares against github.event.pull_request.base.sha. That value is the base commit recorded when the pull request was opened (669f217); it does not follow edge. The job checks out the merge with today's edge (80193c6), so server: comment moderation no longer sends commenter IP addresses to a third-party geolocation service #2961's server/src files count as this pull request's changes, next to its one-step change to the routing workflow. The entry point passes the guard exactly the value the workflow passed before, so the workflow as it is on edge gives the same red for any pull request that touches the routing rules once edge gains a server commit. Locally, against the true base, the guard passes. Two ways out, both the owner's call: re-target the pull request's base (which refreshes the recorded commit), or have the routing workflow pass origin/<base branch> as the other two guards do (a change to that check's behaviour, so not made here).

Also noted

  • The coordinator campaign's known red, delphi/tests/coordinator/test_step2_review_controls.py::test_actual_pca2_route_serves_generation_zero ('"rustproto-0"' == '"0"'), reproduces on the box. Not touched.
  • Test Math: hosted uses JDK 16.0.2 and Clojure CLI 1.10.1.693; make check-math now requires that pair unless the override is set. Clojure dependency resolution is the only network access at test time.
  • make check-client-alpha leaves client-participation-alpha/.astro/types.d.ts modified in the working tree (the Astro build rewrites it). Not touched.
  • The selector test runs in make check-fast only; no workflow runs it.
  • One make check-fast run on the freshly rebooted, loaded box lost a jest worker to SIGSEGV (a Node crash, no assertion); the next run passed. Recorded, not chased.
  • The characterization replay (server characterization: harness runs again on edge; CI replay #2955) is not on edge; its entry point belongs with it.

Verification

  • Every suite above run on the box through its entry point, one at a time and then via make check and make check-fast, on this branch's exact tree (committed locally on the box: the coordinator refuses uncommitted source).
  • Vote-sign lint: 0 new sites; the allowlist is not touched (header 617 = sum of entries). Two-convention gate: v0 green, ratchet green.
  • docker compose -f docker-compose.test.yml --env-file test.env config with no variables set gives the same ports, image names and environment as before.
  • No test assertion, golden, recording or allowlist entry changed. The diff is ci/local/, the Makefile, docker-compose.test.yml (variables with today's defaults), the workflows, .gitignore, and docs.
  • After the review fixes: bash -n on all 28 scripts; the selector test; make check-fast; make check; math three ways (default stop, override, exact hosted pair); server integration again with and without CHECK_STRICT_READY=1; all on the pushed tree.
  • The branch was rebased once onto edge before the first push and since then only merged with edge (669f217) and fast-forwarded.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

colinmegill and others added 9 commits October 4, 2026 19:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… DynamoDB port

With CHECK_PORT_BASE set, the simulator's own healthcheck (port 3000 inside the
container) failed, and the E2E server container still looked for DynamoDB on
host port 8000. Defaults (CI) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…fault

Readiness waits and probes that did not fail the hosted jobs before (the
fixed sleeps and the health probes ending in '|| true') warn and continue
again; starting containers stays fatal. The Compose health wait is bounded
by the old sleep (30 s; 20 s for Delphi). The E2E probe is back on
/api/v3/participationInit. CHECK_STRICT_READY=1 makes readiness fatal.

A suite whose workflow pins a runtime (Node, Python, JDK and Clojure CLI,
the crate's Rust toolchain) now stops with a one-line message when that
version is not available; CHECK_ALLOW_RUNTIME_MISMATCH=1 warns and continues.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ordings entry point

The selector gains the inputs it missed (the two local requirements files,
docker-compose.yml and the copied deploy hooks for Delphi, test.env for the
unit tests, and others). ci/local/test_changed_suites.py reads each suite
script, extracts the repository paths it names and fails when the selector
would not run that suite for one of them; it runs in make check-fast.

Routing recordings get ci/local/routing-recordings.sh and the workflow calls
make check-routing-recordings; its name, triggers, path filters, job, service
container and cache are unchanged. Comments say 'fast loop' throughout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1291 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 178 55%
database/postgres.py 414 95 77%
delphi_storage/init.py 1 0 100%
delphi_storage/codec.py 295 23 92%
delphi_storage/golden_corpus.py 60 9 85%
empty_output.py 29 2 93%
engine_adapter.py 317 265 16%
job_child/init.py 323 54 83%
job_child/census.py 101 13 87%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 8 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 936 32 97%
poller/capacity.py 444 17 96%
poller/capacity_manifest.py 260 13 95%
poller/coordinator_bridge.py 317 317 0%
poller/large_class.py 150 2 99%
poller/math_writer.py 113 3 97%
poller/promotion.py 127 2 98%
poller/readiness.py 368 12 97%
poller/service.py 644 39 94%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 285 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 776 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 284 93 67%
tools/engine_rebuild.py 230 11 95%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 113 38 66%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 434 275 37%
umap_narrative/801_narrative_report_batch.py 837 620 26%
umap_narrative/803_check_batch_status.py 267 148 45%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 234 234 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 331 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 580 466 20%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 379 229 40%
umap_narrative/topic_naming.py 148 148 0%
utils/cli_flags.py 15 0 100%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 159 5 97%
Total 24253 8134 66%

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant