Skip to content

👷 ci: gate coverage on lines nothing ran - #2162

Merged
gaborbernat merged 11 commits into
mainfrom
fix/enforce-total-line-coverage-1745
Sep 9, 2026
Merged

👷 ci: gate coverage on lines nothing ran#2162
gaborbernat merged 11 commits into
mainfrom
fix/enforce-total-line-coverage-1745

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Sep 2, 2026

Copy link
Copy Markdown
Member

--fail-uncovered-lines 0 reads a narrower thing than its name suggests, so the workspace passes it while lines nothing executed sit behind a covered caller. This closes that gap, and it closes it against a number that means what the gate wants rather than against LLVM's own total.

LLVM's total is not the count of lines nothing ran. It sums a file's lines per function, and where a generic has several monomorphizations it scores their instantiation group by taking the mapped count and the covered count from whichever member is highest at each, on its own. A line one monomorphization runs and another does not then counts as missed while executing. On this workspace the total reported 88 lines and this check reports 22, so 66 of the 88 were reported missed while executing. Closing those would have meant changing which concrete type a test picks, which moves the count alone.

A reviewer can confirm that rather than take it. The old flags and the new check ran in the same job on the same profile data: the total said 271519 lines and 88 missed, and coverage_lines.py names 22 in 15 files. The classification behind the split came out of #2232, which read the per-function line stats out of the JSON export, grouped them by instantiation, and closed eighteen lines that were genuine gaps; this check is that method wired into the gate.

coverage_lines.py takes the union across a group: a line stays missed only when no member of its instantiation group ran it. That leaves the gate on unexecuted code and drops the merge artefact, so it measures the truth rather than loosening the bar. --fail-uncovered-lines 0 stays exactly as it was, because it is the half that finds real gaps: every line it named today was a genuine defect or a missing assertion, across #2213, #2187, #2195 and #2231. What it cannot see is a function body nothing ran whose lines a covered caller spans, and that is what the new check adds.

_coverage-lines-contract makes the behaviour checkable. It builds a two-file export where one file holds two monomorphizations of a generic, each running the line the other does not, and the second holds a body nothing ran, then asserts the first is forgiven and the second reported. Scoring the group by its best single member, the way LLVM's total does, makes that contract fail; so does reporting nothing at all.

Worth stating as the boundary of what any of this proves: passing a line gate does not mean a crate has no unreachable code. A line that ran can still hold an arm that never did. The pypi lane found page_stream/mod.rs:335, a size-limit guard that can never match, because stream_detail returns Fallback at line 125 before its only call to streaming_parts. Line coverage called it covered, since the guard and its Ok(None) body share one line and evaluating the guard marks the line executed. Mutation testing caught it when line coverage could not. Both facts follow from measuring lines rather than branches: one direction demands edits for lines that already execute, the other certifies lines whose dead arms it cannot see.

On this branch the gate named 14 lines in 10 files, and all 14 are closed here. Two were fallibility no run can reach, so they are gone rather than tested. plan already turns a second snapshot into RetentionPlanError::Store before the callback export_body installs is called again, which made the guard inside that callback a second copy of the same rule carrying the same message. serde_json::to_string over a Value cannot fail, so the retention CLI header takes the same non-fallible path as the decision writer beside it.

One was an assertion that could not fail: the oci admission test searched an empty capture for a warning, which passes whatever the push did, and it now compares the whole captured set against the key the push staged under. The other eleven are error arms with a fault driven through them. A truncated multipart body reaches the two field readers in upload_form.rs. Injection sweeps across the redb backend reach store/files.rs:140 and both filter scans in store/projects.rs, and a store whose policy generation row does not decode reaches the retention plan and the metadata rebuild in the peryx fixture driver. A replication token the leader answers 401 to reaches the forwarded ClientWrite in consensus_runtime.rs. A startup line carrying no port reaches the bench harness, and two fixture modes reach the second startup wait and the status probe in peryx-test-support.

store/files.rs:140 was listed above as resisting, because a redb handle does not survive its own injected failure and an Err cannot be told apart from a commit that landed and then failed its sync. That distinction turns out not to be the one worth making. The property is that the sweep leaves every legacy row or none, which holds at both ends of that range and rules out the half-swept store an operator reads as progress.

Closes #1745

@gaborbernat gaborbernat added the enhancement New feature or request label Sep 2, 2026
@codspeed-hq

codspeed-hq Bot commented Sep 2, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 29 untouched benchmarks
⏩ 133 skipped benchmarks1


Comparing fix/enforce-total-line-coverage-1745 (c4815b4) with main (34eca99)

Open in CodSpeed

Footnotes

  1. 133 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@read-the-docs-community

read-the-docs-community Bot commented Sep 8, 2026

Copy link
Copy Markdown

@gaborbernat
gaborbernat enabled auto-merge (squash) September 8, 2026 16:08
@gaborbernat gaborbernat changed the title 👷 ci: hold native coverage to LLVM's total 👷 ci: gate coverage on lines nothing ran Sep 8, 2026
@gaborbernat
gaborbernat force-pushed the fix/enforce-total-line-coverage-1745 branch from 2413eb9 to 66ddcb4 Compare September 9, 2026 10:26
`--fail-under-lines 100` compares against LLVM's own line total, which is not
the number of lines nothing executed. LLVM sums a file's lines per function and
scores an instantiation group by taking the mapped count and the covered count
from whichever monomorphization is highest at each, independently, so a line one
monomorphization runs and another does not is counted missed while executing.
That described 66 of the 88 lines the total reported on this workspace, and
closing them would have meant changing which concrete type a test picks.

The check takes the union across a group instead: a line stays missed only when
no member of its instantiation group ran it. That keeps the gate on unexecuted
code and drops the merge artefact, so it measures more accurately rather than
more permissively.

`--fail-uncovered-lines 0` stays as it was. It is the half that has named
genuine gaps, and the new check covers what it cannot see, which is a function
body nothing ran whose lines a covered caller spans.
`lint-automation` runs the contract that exercises the line check, and that job
declares no `uv`, so the contract died on a missing command rather than on
anything it asserts. The check now runs under the interpreter the runner image
carries, the way these recipes already reach for `jq`, so neither job needs a
tool it does not declare.

`_coverage-target-contract` asserted the flag string the gate used to carry. It
now asserts the two halves that gate the run and that LLVM's own total is not
one of them, which is the property rather than the spelling. Naming a flag in a
recipe comment is enough to satisfy a grep over the dry run, so the comment
beside the gate no longer spells it.
Four dashboard pages render their loader's error only under `hydrate`, where the
fetch behind it can fail: every `ssr` loader returns `Ok` with no fallible step,
so the six closures cannot run in the native build. Measured in the frontend
tracefiles, all six run in the Wasm target with counts of 1 to 10 and none in
the native one.

Reporting them missed because only one target was looked at is the same error as
counting a line one monomorphization runs, a layer up. `coverage-native` now
writes its gap list instead of failing on it, and `coverage-lines` fails on what
is left once the browser target's tracefiles are unioned in. That drops 22
reported lines to 16.

Forgiveness goes by function start rather than by line hit. A line hit says
something on the line ran, which a dead arm sharing a line with its covered
guard would satisfy; a function beginning on the line having run is the claim
these six actually support.

The two artifacts are small, so a job waiting on both adds a minute rather than
serialising the two long ones.
The line gate this branch adds reports one line for the whole workspace:
`server.rs:781`, the `with_context` closure on a failed `webhook_events`
lookup. Nothing that calls `build_webhooks` from outside can reach it. The
registry fails that lookup only for an ecosystem it does not hold, and both
`activate_plugins` and `build_index_settings_with_plugins` reject such an
index before the webhook loop runs, so the arm belongs to the private
function rather than to any entry point.

The test therefore calls `build_webhooks` through a `#[cfg(test)]` `#[path]`
module inside `server.rs`, which reaches the private function without making
it visible to the rest of the crate. An index that names an uninstalled
ecosystem and carries one webhook is what drives the lookup, and the
assertion pins the whole chain, context and source together, so a message
that stops naming the index fails here.

Line 781 now reports a count of two, and the crate's 1099 tests pass.
Both arms sat on the export path and neither could ever be taken, which the
line gate reports as code no run executes.

`plan` already rejects a driver that opens a second snapshot, and it does so
before the callback `export_body` installs is called again, so the duplicate
guard inside that callback is a second copy of a rule enforced one layer up.
The two copies even carried the same message. Keeping the check where the
caller can still report it and asserting the invariant here says which layer
owns it.

The CLI header took the other shape: `serde_json::to_string` over a `Value`
cannot fail, so the error it mapped was unreachable. The decision writer beside
it already serialized without a `Result`, so widening that helper to any record
lets the header take the same path instead of hand-rolling a fallible one.
Two error arms inside the fixture ecosystem driver had no caller: the write
failure a metadata rebuild reports, and the read failure a retention plan
reports when the policy generation cannot be decoded. Both are arms the real
drivers carry too, so leaving them unrun leaves the app's handling of them
unrun as well.

A rebuild and a preview write the same records through the same handle, so the
output-failure test covers both by case rather than by a near-duplicate beside
it. For the generation read there is no need to fault the backend: a row that
does not decode fails the same read, and asserting the command wrote nothing
past its header pins that the plan stops before it invents a frontier.
The below-threshold admission test searched an empty capture for a warning,
which passes whatever the push did and would pass if admission never ran at
all. Comparing the whole captured set and the key the push staged under says
both that nothing was logged and that there was a push to log about.

The bench harness reads a server's real port out of its startup line, and the
arm that reports a line carrying no port had no test. A fake server that
announces itself and says nothing else drives it in a few milliseconds, so the
arm is covered without a benchmark run.
Four error arms had no caller. Two read a multipart field, one sweeps legacy
source rows and one filters the project listing, and each is the arm that
decides what a client or an operator is told when the read behind it stops.

A body that ends inside a part is a different failure from one that ends in the
part headers, which is what the existing test covers: the field reader is
already streaming when the stream ends. Both readers meet it, so the recognised
field and the drained one run as cases of one test.

The store arms are driven by sweeping the injection point across the backend
rather than targeting one read, so the tests carry no magic constant and assert
a property that must hold at every point: the sweep leaves every legacy row or
none, and the listing answers with the retirement applied or not at all. A
commit that lands and then fails its sync is not a partial sweep, so both ends
of that range are accepted and the middle is not.
A follower forwards an ownership write to the leader over its own replication
token, and the transport failure that forwarding can meet had no test. The arm
decides what a caller is told when the write never reached consensus, which is
different from being told it went to the wrong node.

A token the leader does not hold is the deterministic way to make that call
fail against a live group: the peer answers 401 without any node leaving or any
election running, so the follower still names the same leader and the test has
no timing to lose.
@gaborbernat
gaborbernat force-pushed the fix/enforce-total-line-coverage-1745 branch from 414c7d7 to f8f6cf8 Compare September 9, 2026 17:09
The harness waits for the startup signal twice and then probes the status
endpoint, and the failure of the second wait and of the probe had no test. Both
decide what a system test is told when its node never comes up, so a wrong
answer there is read as a fault in whatever the suite was testing.

Two fixture modes construct them without any timing. One emits a second event
that is not the startup signal, so a process reaped between the two waits
leaves the second one holding a child it can no longer collect. The other
announces itself and leaves; the parent keeps the port bound, so the probe that
follows reaches a socket with nothing behind it rather than a closed one.

A reap ends either wait the same way, so the reap test reads the log to say
which one ran. The second mode also needs a test whose process leaves through
its shutdown request: a killed child writes no profile data, so the events it
printed are recorded only by a run that exits. That test asserts the message
the timed-out second wait reports, which is what tells a reader it heard
everything except the signal rather than nothing at all.
The consensus-teardown ordering test dropped the runtime handle without
waiting for the listener task to be polled, so the future it was there to
observe could still be waiting for its first poll when the cancellation
arrived. It passed either way, because the signal it reads comes from a
destructor that runs whether the future completes or is dropped.

What it lost was the ordering it claims to check, and the tell was coverage:
the body of the listener future stopped executing under a load the run before
it had not put on the box. Handing the listener the same entered handshake its
sibling double already uses puts the poll before the drop, so the teardown the
test names is the one it observes.
@gaborbernat
gaborbernat force-pushed the fix/enforce-total-line-coverage-1745 branch from f8f6cf8 to c4815b4 Compare September 9, 2026 17:42
@gaborbernat
gaborbernat disabled auto-merge September 9, 2026 18:39
@gaborbernat
gaborbernat merged commit ceecc88 into main Sep 9, 2026
21 of 23 checks passed
@gaborbernat
gaborbernat deleted the fix/enforce-total-line-coverage-1745 branch September 9, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce LLVM total line coverage

1 participant