Add mutation testing on the files each pull request changes - #82
Merged
Conversation
A passing test suite does not show which faults it can detect. This adds a per-revision mutation run that produces that evidence for the files a pull request actually touched. Scope comes from git, not from a budget: the workflow diffs merge-base..head, selects production sources in each cohort, and passes them to Stryker, which has no changed-files selection of its own. A revision touching no production source is reported as not_applicable, which is neither a pass nor a failure. Stryker is treated as a black box that produces observations, never verdicts. Its raw statuses are retained and re-projected through a conservative table: a Killed without retained assertion output is inconclusive rather than a kill, a timeout is not a kill, and uncovered code is not a survivor. Inconclusive trials stay outside the killed+survived denominator. The client cohort runs on the Vitest runner; the reference implementation runs on the command runner with coverageAnalysis off and test selection inside the command, since that runner reports no individual test identities. Incremental reuse is keyed on recorded execution inputs -- config, tool version, runtime, and lockfile digests -- so a change Stryker does not track cannot reuse stale verdicts. There is no age or cost threshold. The job publishes evidence as artifacts and does not block merge. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…ills Three defects, each found by running the code rather than reading it. The reference-implementation cohort could never run. Its configuration reads the tests to run from a file that nothing wrote, so it always took a fallback of `node --test test/` -- and Node resolves a bare directory as a module, not as a test glob, so the baseline failed with MODULE_NOT_FOUND on every applicable revision. Because the run step tolerates failure and a rejected baseline makes every mutant inconclusive, that cohort reported success while producing killed=0, survived=0 and no evidence at all, forever. The intent stage now derives the test selection from the same diff that produces the mutate list and writes it, and the fallback is a glob Node expands. An applicable run that produced no evidence now fails. The receipt was already honest -- it recorded "no evidence" accurately -- but a green check on top of an empty receipt is not, and the check is what a reader sees first. A rejected baseline, a run that wrote no report, and a batch in which every trial was inconclusive each exit non-zero, with the receipt and raw report still published. Survivors do not fail the job; only the absence of evidence does. The incremental cache was keyed so it could never hit. The key hashed the whole intent packet, which contains the head commit and the changed-file list, so it was unique per revision and every run was cold. It is now keyed on the digest of the execution inputs alone. `mayReuseCache` was correct and tested but unreachable from the workflow, which trusted the cache action's own hit output instead; a new step calls it, deletes a cache whose recorded inputs do not match, and records that verdict in the receipt. Kills are decided from structure, not from failure prose. The predicate regex-matched failure text for assertion vocabulary, and on the first client file it ran against it refused five of six real kills: those mutants made a validator reject a valid input, so a ZodError was thrown inside the subject expression of a genuine assertion and the retained text was a Zod issue array with no assertion words in it. A dialect list cannot fix that, because the text of a real assertion kill is whatever the thrown value happened to print. A kill now requires a killing test identity from the runner's own per-test results and the absence of crash evidence. Re-projecting the same report gives killed=6 of 6 rather than 1 of 6, and uncovered code is still never a survivor. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak
force-pushed
the
test/reactive-mutation
branch
from
September 9, 2026 17:41
d30f956 to
c0e22ba
Compare
…he attempt The reference-implementation cohort mutated zero files. Every relative path it uses is cohort-relative -- the mutate globs, the configuration's reporter path, its test-selection lookup and its sandbox exclusions -- and Stryker resolves all of them against the working directory rather than against the configuration's own location. The workflow invoked it from the repository root, so the globs matched nothing, the sandbox tried to copy the whole repository, and the report was written where the next step does not read. Running it from the cohort root settles those at once: the same revision now instruments one file into two mutants and writes its report exactly where the projection step looks for it. A failed attempt and a clean attempt that produced nothing wrote byte-identical receipts. Both have zero trials, so the projection alone cannot separate them, and the check exiting non-zero does not help a reader holding only the artifact. The receipt now carries the engine's exit status, whether a report was written at all, whether a baseline was established, and whether the attempt produced evidence -- so the record that outlives the run says what happened rather than leaving it in process output. Deletions never reached the classifier. The diff was filtered to added, copied, modified, renamed and type-changed paths, which dropped deletions before the selector saw them, while the selector's own comment promised the exclusion list was a full account of the diff. Deletions are now included and recorded as excluded, which is what that list is for. A cohort's own test files were labelled as being outside the cohort. Cohort membership was decided before asking whether a path is a test, and the reference implementation keeps its tests outside its production prefixes, so every one of them got the wrong reason. Both reasons exclude, so no mutate list ever changed; only the recorded reason was wrong. Also removes the job's duration cutoff, which contradicted this workflow's own statement that it holds no duration constants, and restores the plain explanation of why three local tooling directories are excluded from the sandbox copy. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
The incremental cache had two independently reproduced defects, and removing it fixes both without adding a repair layer. The reference implementation read and wrote its incremental report beneath its own root, because Stryker resolves that path against the working directory and this cohort runs from its own directory. Every mechanism that managed the file -- the cache save and restore, and the step that refused a cache whose inputs had changed -- used the repository root instead. So the cohort's cache could never be stored, and a refusal deleted a copy the engine does not read while leaving the one it does. A run could reuse a cache the record said was refused. The identity that authorised reuse was also incomplete. It covered the configuration file's bytes, the tool and runtime versions and the lockfile, but not the resolved test command. Two runs executing different test files hash identically, so the identity would have approved reuse across runs that did not run the same tests. Neither defect is needed to run mutation testing on the files a pull request changed and retain its evidence, and a cache is an optimisation rather than a result. So the cache is gone: the two workflow steps, the decision entry point, the reuse predicate, the recorded cache decision in the receipt, and the incremental settings in both configurations. Every run is now cold. That costs time and removes any possibility of reusing a stale verdict. Also guards the change of directory before the engine runs, so a failure there cannot silently mutate nothing from the wrong place, and corrects a comment that claimed that change of directory settled the incremental-file path. It did not. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak
force-pushed
the
test/reactive-mutation
branch
from
September 10, 2026 01:39
4e5f47c to
21ec75e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A suite that always passes proves nothing about whether its assertions still bite. Mutation testing introduces small faults into production code and checks that a test fails.
This adds Stryker on every pull request, scoped to the production files the PR changed between merge base and head. A PR that changes only tests or docs mutates nothing and reports
not_applicable. The client cohort uses Stryker's Vitest runner. The reference implementation uses the command runner withnode --test, coverage analysis off, and test selection inside the command. Stryker runs from each cohort's own root, because every relative path in its configuration resolves against the working directory.Stryker's own
KilledandSurvivedare recorded verbatim but not used as the verdict. A separate projection inscripts/mutation-falsification/stryker-adapter.tscounts a kill only when a named test failed, and keeps crashes, timeouts, and no-coverage as inconclusive. The denominator is killed plus survived; inconclusive mutants never enter it.There is no result cache: every run is cold, so no run can reuse a stale verdict. Each run uploads three records as PR artifacts, including on failure: the frozen intent, the raw Stryker output, and the projection.
Verify:
npx vitest run scripts/mutation-falsification, then run the workflow's script locally against one changed file as.github/workflows/mutation.ymldoes.Not established: a run on the applicable path in GitHub Actions itself, since this PR changes no production source.
Assisted-by: AI