Probe the audit store's capacity and report it on the endpoint (#774)

[sehkone]

Part of #774
Part of #775

#774 is the issue this pull request delivers; #775 is the umbrella it hangs under, and the reference to it is inherited through #774 rather than being work this pull request implements directly.

What this adds

An endpoint-enabled daemon now measures the reserved audit store on the rotation loop's existing maintenance tick and reports what it found in every response's registrar_health.audit_capacity member — on refusals as well as successes, because the store is a fail-closed control and a success-only signal would stop carrying the alarm in exactly the state the alarm exists to announce.

The probe — src/registrar/audit_store/capacity.rs (new)

Both halves of the headroom come from one O_NOFOLLOW | O_DIRECTORY | O_RDONLY | O_CLOEXEC open of audit_store_dir, in both enforcement modes. A pathname statvfs beside a separately opened walk root resolves the configured path twice and nothing holds the two resolutions to the same directory — and it would quietly report a planted symlink's target. A link at the store root is an ELOOP from that open and a failed probe in both modes.

Usage is measured per mode, because statvfs describes a filesystem and not a subtree:

• filesystem — (f_blocks - f_bfree) * f_frsize from the same fstatvfs on the same descriptor. Nothing is enumerated.
• directory — the store is walked, summing st_blocks * 512.

Available bytes are f_bavail * f_frsize in both modes, never f_bfree, which counts blocks reserved for root that neither writer can reach.

The walk descends by descriptor — openat/fdopendir/readdir/fstat/fstatat/closedir — rather than by path, because openbao/ is written by the OpenBao container's uid by design, so a directory replaced under an active walk is an attacker-reachable event and not a hypothetical one. It skips . and .. by name before any fstatat, openat or accounting; never opens a non-directory; stays on one device; counts each (st_dev, st_ino) once; counts each directory's own blocks including the store root; distinguishes end-of-directory from a readdir error by zeroing errno first; and fails the probe on every error except an entry that vanished between readdir and the call that followed it, which is the ordinary rotation race.

The descriptor has exactly one owner at every point. fdopendir receives it by into_raw_fd(), a null return reconstructs the OwnedFd and closes it before propagating, and after a successful fdopendir closedir is the only close. Every unsafe block wraps one call and carries a // SAFETY: comment.

This is deliberately a second implementation of "sum a subtree's allocated blocks". measure_underlying in src/commands/audit_store/reserve.rs is unchanged and the bootroot init reserve preflight is not rewired: that one is path-based, runs once under an operator's init on a quiet store and fails on arithmetic overflow, while this one runs unattended every minute against a live, partly attacker-writable subtree and saturates instead.

The alarm

headroom_bytes = min(reserve - used, available), computed with checked conversions and an explicit clamp — no as cast between the u64 inputs and the i64 result, and every block-count product saturates at u64::MAX rather than wrapping or failing the probe. next_state applies the three ordered rules exactly: non-positive headroom is exhausted; a return from an alarm clears only at threshold + margin; otherwise ok strictly above the threshold. margin = max(low_water / 10, 1 MiB), and the clear sum is a saturating add.

The wiring

run_rotation_loop_with_maintenance's callback becomes async, awaited in the same tokio::select! branch body that already awaits rotation.run_pass(...). The capacity probe and scan_audit_store_off_runtime both run under spawn_blocking with their handles awaited inside the tick. No second scheduler, interval or long-lived task; no new health holder; ROTATION_INTERVAL unchanged, so measured_at and records_measured_at are never more than about a minute old in a healthy daemon.

The three record signals are read from scan_audit_store with AUDIT_SCAN_WINDOW and never re-derived, so the relayed values and what bootroot status prints host-locally cannot drift.

On the response side, all three arms — mint success, deregistration success and refusal — take the snapshot from one private accessor, ProductionHandler::health_snapshot, so the request path has a single source for it and nothing else reaches the holder.

The wire

RegistrarHealth gains audit_capacity, appended after limiter, whose shape, contents and ordering are untouched. Optional members are omitted rather than emitted as null and an explicit null is refused on decode; both timestamps are RFC 3339 UTC Z strings. AuditStoreEnforcement gained Serialize rather than a second enum being declared for the wire. All five golden fixtures that carry the container were extended additively; the pre-registrar audit_unwritable refusal still serializes "registrar_health":{}.

Configuration

audit_store_low_water_bytes = 0 is now refused at load with a diagnostic naming the key: at zero the low-water band is empty, so the state machine would step straight from exhausted to ok and the alarm this key exists to raise could never fire. No key was added and no other key's validation changed.

Documentation

A new "Watching the reserve fill" section in docs/en/operations.md and docs/ko/operations.md covers the thresholds, the four state values, the hysteresis, the two timestamps and what bounds their staleness; the audit_store_low_water_bytes entry in both configuration.md pages is updated; docs/reference/registrar-wire-contract.md documents the audit_capacity schema beside the limiter paragraph. No mkdocs.yml nav change.

Test plan

Derived from the issue's acceptance criteria and test plan. Every box below was run and passed unless the line says otherwise.

Capacity, alarm and arithmetic (src/registrar/audit_store/capacity/tests.rs)

• The alarm is on at headroom_bytes == audit_store_low_water_bytes, off at low_water + 1 from a cold start, does not clear between the threshold and low_water + margin, and clears at low_water + margin — driven through the probe abstraction over a tempfile::tempdir().
• A second test drives the real fstatvfs-backed probe against a tempdir and asserts plausible non-zero values.
• state table: unknown before any probe, ok, low_water at and just below the inclusive threshold, exhausted at zero and at negative headroom, the hysteresis-gated return to ok, the immediate exhausted → low_water step on the first positive headroom below the clear threshold, the direct exhausted → ok step when one probe reaches threshold + margin, and a failed probe leaving the previous state and measured_at intact.
• The margin is max(low_water / 10, 1 MiB): the floor applies where the 10% term truncates to zero, the 10% term applies at the default, and a threshold near i64::MAX saturates the clear sum rather than wrapping it.
• Headroom conversion saturates: used_bytes and filesystem_available_bytes above i64::MAX yield neither a spurious ok nor a spurious exhausted; f_bavail × f_frsize, (f_blocks − f_bfree) × f_frsize and st_blocks × 512 each saturate to u64::MAX with the probe still succeeding.

Usage measurement and the walk (real tempfile::tempdir() fixtures)

• filesystem mode derives usage from the same fstatvfs on the store root descriptor and performs no walk; directory mode walks the store and tracks a file written into it; available bytes are f_bavail-derived in both.
• Entry types: a symlink to a large outside file and a symlink to a directory add only their own entries; two hard links to one file count its blocks once; a FIFO is counted from its own fstatat and neither opened nor traversed; a sparse file counts by allocated blocks; a nested real subdirectory on the same device is descended and its own blocks counted. The expected total is computed from each fixture's own reported st_blocks.
• Dot entries: with a large file and a populated subdirectory beside the store in its parent, the reported usage is the store subtree's own total and is unmoved when those parent-side bytes grow.
• A symbolic link at audit_store_dir itself fails the probe in both enforcement modes and leaves the previous state and measured_at intact.
• A walk that cannot read part of the store fails the probe rather than summing the unreadable part as zero.

The walk's test seam (failure injection only; every other walk test runs against a real tempdir)

• ELOOP/ENOTDIR at a descent openat fails the probe and contributes no usage at all.
• ENOENT at the openat/fstatat following a readdir is skipped and the probe still succeeds.
• EACCES on a subdirectory fails the probe and leaves the previous state and measured_at intact.
• A null readdir return with a nonzero errno (EIO mid-directory) fails the probe rather than reading as end-of-directory, and the short total is not reported.
• A null fdopendir return after a successful openat fails the probe and closes the descriptor its openat produced.
• A failed fstat on an enumerated subdirectory's own descriptor fails the probe — the call that proves a directory is classified from the descriptor the walk traverses, not from the fstatat that decided to open it.
• Descriptor hygiene: open and close counts balance for a successful walk, for one aborted before the ownership transfer, and for one aborted by an injected fdopendir failure after it — with no /proc/self/fd read, no process-global count and no fcntl check.

The tick (src/daemon/audit_capacity_tests.rs)

• The probe and the scan run on the existing rotation-loop maintenance tick at ROTATION_INTERVAL; no second scheduler, interval or long-lived task is added, both filesystem operations run under spawn_blocking, and every spawned handle is awaited inside the tick.
• The relayed record values equal scan_audit_store's own output for the same store and window, and the tick declares no window value of its own — asserted over its source, so neither a second constant nor an inline 30-day duration can creep into the call site.
• The exact malformed_records count reaches the health response and bootroot status alike, and a malformed line in a surplus rotated generation the reader does not select is counted by neither. No assertion is made about a malformed line's age.
• The retention shortfall reaches both surfaces in both directions — true for a forced store, false for a healthy one.
• A failed probe leaves the previous state and measured_at intact while the scan still succeeds, and a failed scan leaves the previous three record values and records_measured_at unchanged rather than zeroing them.
• Before the first successful scan the four record members are absent while the capacity members are present.
• enforcement is always present and mirrors the configured mode in both deployments.

The wire (src/registrar/endpoint/protocol.rs, src/registrar/endpoint/tests.rs)

• registrar_health.audit_capacity carries every specified member with the specified types and presence rules — state and enforcement enums, headroom_bytes signed, both timestamps RFC 3339 UTC, the three capacity measurement members absent exactly when state is unknown, the four record members absent exactly before the first successful scan.
• A response carrying the member beside limiter round-trips, and no limiter byte changed.
• An optional member is omitted rather than emitted as null, and an explicit null in a decoded payload is rejected — both directions, for one optional member of each type.
• The pre-registrar audit_unwritable refusal path still serializes "registrar_health":{}.
• All five golden fixtures carrying the container were extended additively: mint-success.json, deregister-success.json, refusal-permanent.json, refusal-busy.json, refusal-unclassified.json.
• A response relays the last snapshot and performs no store scan of its own — over the deregister and refusal shapes end to end, and over the mint shape in production::tests, where the handler's own health_snapshot() feeds the production mint encoder and the reader is then run against the handler's own store in the same test and returns a different count from the one the response carried.
• No arm of the request path can scan at all: no_request_path_arm_reads_the_audit_store reads this build's mint and deregister bodies and asserts each takes its health from the single accessor, reaches the holder through nothing else, and that the module names no store reader.

Configuration and provenance

• audit_store_low_water_bytes = 0 is rejected at load with a diagnostic naming the key, and the existing reserve and upper-bound rules still reject what they rejected before.
• measure_underlying in src/commands/audit_store/reserve.rs is unchanged and the bootroot init reserve preflight is not rewired.
• Tests use tempfile::tempdir() and never a fixed path, and mutate no process environment.

Suites and gates

• cargo test --bin bootroot — 1286 passed on the macOS host; 1305 on Linux. The bootroot status agreement tests live in the binary crate, so cargo test --lib alone would skip them.
• cargo test --lib — 1068 passed on macOS, 1349 in a Linux container as a non-root user. src/registrar/endpoint is gated on target_os = "linux", so the endpoint tests run only in the container; the capacity module carries #[cfg(any(target_os = "linux", test))] and its own tests run on both.
• cargo test --no-fail-fast — every target green on macOS. In the Linux container two unrelated targets fail for reasons that are the container's: tests/bootroot_rotate.rs's eight rotate ca-key cases need a docker binary it has none of (all 46 pass on macOS), and tests/bootroot_verify.rs's test_verify_success fails only under that container's full-suite parallelism and passes alone. Neither touches anything this change adds.
• cargo fmt -- --check --config group_imports=StdExternalCrate, cargo clippy --all-targets -- -D warnings, and cargo doc --no-deps --document-private-items with RUSTDOCFLAGS=-D warnings — the new module, its trait, its enums and the new health member all carry rustdoc. Clippy and rustdoc were additionally run on Linux, where the endpoint and the daemon's registrar wiring actually compile.
• markdownlint over the changed Markdown and ./scripts/check-docs.sh for the docs/ changes; scripts/preflight/ci/check.sh as a whole, including ruff, biome and cargo audit.
• scripts/validate-deploy-compose.sh, validate-compose-instance-names.sh, validate-e2e-openssl-compat.sh, validate-e2e-leftover-check.sh, validate-e2e-run-scope.sh.
• Documentation: the thresholds, the state values, the hysteresis, the two timestamps and what bounds their staleness are covered in both docs/en/operations.md and docs/ko/operations.md, audit_store_low_water_bytes is updated in both configuration.md pages, and docs/reference/registrar-wire-contract.md documents the audit_capacity schema. No mkdocs.yml nav change.
• scripts/preflight/ci/test-core.sh and scripts/preflight/ci/e2e-matrix.sh — did not run on this machine. See Preflight.

Preflight

scripts/preflight/ci/check.sh and every validate-*.sh above ran and passed on this machine.

scripts/preflight/ci/test-core.sh and scripts/preflight/ci/e2e-matrix.sh did not run, for two reasons that are properties of this machine and not of the change:

1. The matrix's step 13 runs bootroot init as root through sudo -n, and sudo -n on this host fails with "a password is required". --skip-hosts does not stand in for it.
2. Ports 8200 and 9000 are held by an unrelated live bootroot-* Compose stack belonging to another session on this host. Both scripts begin by tearing down that Compose project, which would destroy work that is not mine.

Nothing was weakened, skipped, marked continue-on-error or deleted to make a local run pass. This change touches daemon scheduling, configuration and the endpoint, so it is not eligible for the E2E exemption: CI's Docker E2E jobs gate the arm that did not run here, including the endpoint-enabled loopback init in step 13 that exercises the very filesystem-mode measurement this adds.

Not addressed

• The mint arm's relay is asserted over the two halves it is made of, not through one handle round trip. Operation::Mint is refused at requested_spec with no response bytes — minting waits on the wire spelling of a request's spec, which this repository is explicitly not the author of — so no payload reaches the mint encoder through handle. The regression test therefore drives ProductionHandler::health_snapshot(), which is where the mint arm takes the value, into protocol::encode_mint_response, which is what that arm hands the value to, and a companion source-level test pins that the arm calls exactly those two and nothing store-reading. Together they cover the guarantee the round trip would have; only the frame around it is missing.

• bootroot status and the health response are asserted to agree by each being held against the reader's own output for the same store and window, rather than by one test driving both surfaces over one store. The two live in different crates — status is in the binary crate, the health tick in the library — and a single test cannot reach both. Since both call scan_audit_store with the same five arguments, equality with the reader is the strongest available form of the agreement.

• scripts/preflight/ci/e2e-matrix.sh did not run at all, rather than running as far as it goes. Its first step tears down the bootroot-* Compose project, and on this host that project is a live stack belonging to another session, so there was no prefix of the matrix that could be run without destroying someone else's work. scripts/preflight/ci/test-core.sh is blocked the same way. CI's Docker E2E jobs gate that arm.

[sehkone]

[Reviewer Round 1]

Changes requested.

• src/registrar/endpoint/production.rs:115 — the required regression test that a successful mint response relays the last snapshot without scanning the store is missing. The PR explicitly leaves this unchecked; the mint golden fixture only tests the codec, not the production mint path. This leaves the issue’s no-per-request-scan guarantee unprotected for mint responses.

• PR hygiene: PR #965 begins with both Part of #774 and Part of #775. Per the issue’s inherited linkage, only Part of #775 is expected. Remove the direct Part of #774 reference; the documented ## Not addressed section then correctly describes a partial delivery.

[sehkone]

[Review Verdict Round 1: NOT_APPROVED]

[sehkone]

[Author Round 1]

Pushed as 996e0894; PR description updated.

Review item dispositions

1. src/registrar/endpoint/production.rs:115 — missing mint-path regression test — Partially addressed

Valid, and the gap is now closed as far as this build permits. The reviewer is right that the guarantee was unprotected for mint: the golden fixture only proves the codec carries the member.

What I could not do: drive a mint round trip. requested_spec refuses every wire spec with GrammarNotSettled, and mint() calls it before anything else — so handle(Operation::Mint, …) returns HandlerRefusal with zero response bytes. Making it succeed would mean inventing the spec.reload / spec.cert_group spelling, which production.rs and docs/reference/registrar-provisioning-config.md §3.2 explicitly rule is not this repository's to author; and the alternative — a #[cfg(test)] injection slot — is forbidden by the type's own module doc and pinned by an existing test.

What I did instead:

• Extracted ProductionHandler::health_snapshot() (production.rs:115). All three arms — mint success, deregistration success, refusal — now take the snapshot there and nowhere else.
• a_mint_response_relays_the_last_snapshot_without_scanning_the_store (src/registrar/endpoint/production/tests.rs) drives the two halves the mint arm is made of: the handler's own health_snapshot() into protocol::encode_mint_response, then decodes the mint shape and asserts the daemon-held snapshot verbatim, limiter included. The proof that no scan ran is not that the numbers look untouched — the test seeds a malformed line into the handler's own store, runs scan_audit_store over it in the same test, and asserts the reader returns 1 where the response carried 7. It then mutates the holder and asserts the next response follows it.
• no_request_path_arm_reads_the_audit_store reads this build's mint and deregister bodies and asserts each contains let health = self.health_snapshot();, contains no .lock(), and that production.rs names no store reader. This is the test that actually catches a scan added to the mint arm — which, on this build, no round trip could.

The sibling test in endpoint/tests.rs had its doc comment updated to point at both rather than at the fixture. ## Not addressed now records what remains: only the handle frame around the mint arm, not the guarantee.

2. Remove Part of #774 from the PR body — Pushed back

Not applied. #774 is the issue this PR implements; #775 is the umbrella it hangs under. Removing the direct reference would leave the PR linked only to the grandparent and orphan its own issue. The repo convention is both, e.g. PR #962 opens Closes #768 / Part of #770. Part of #774 rather than Closes #774 is precisely the right form given the ## Not addressed section — the closing keyword is what a partial delivery must avoid, not the reference. Both declarations are unchanged and no other Part of was added.

Verification

Run in a Linux container as a non-root user, since src/registrar/endpoint/* is target_os = "linux"-gated and never compiles on this macOS host:

• cargo test --lib — 1351 passed, 0 failed (14/14 in registrar::endpoint::production::tests, including both new tests)
• cargo clippy --all-targets -- -D warnings — clean
• cargo fmt -- --check --config group_imports=StdExternalCrate — clean
• RUSTDOCFLAGS=-D warnings cargo doc --no-deps --document-private-items — clean
• cargo check --all-targets on the macOS host — clean

No documentation or CHANGELOG change: this round is test coverage plus a private accessor extraction, with no user-observable behaviour change. The preflight limitation on this host (no passwordless sudo, ports 8200/9000 held by another session's live stack) is unchanged and still recorded in the PR body.

[sehkone]

[Reviewer Round 2]

Changes requested.

• PR Probe the audit store's capacity and report it on the endpoint (#774) #965 body — Part of #774 remains at the top. This is still a linkage defect under the review contract: only Part of #775 is expected here. The author’s general convention rationale does not override that explicit inherited-linkage rule. Remove the direct Part of #774 line.

The prior mint-path test concern is resolved: the shared health_snapshot() accessor is used by both response arms, and the new relay/source-guard tests cover the currently unexercisable successful mint frame without introducing a fabricated wire-spec grammar. I found no additional code findings in this round.

[sehkone]

[Review Verdict Round 2: NOT_APPROVED]

[sehkone]

[Author Round 2]

Verified with a re-read: the body now carries one direct declaration for #774, with #775 as prose.

Review item dispositions — Round 2

1. PR #965 body — remove the direct Part of #774 line, keep only Part of #775 — Partially addressed

What I changed. The body no longer carries two stacked bare declarations. Part of #775 is gone as a standalone declaration; the umbrella is now rendered as prose:

Part of #774

That issue is itself part of the #775 umbrella, so this work is inherited by that umbrella too.

That removes the actual defect the linkage rule guards against — a pull request making a direct Part of declaration for an issue it does not implement. #775 is not this PR's issue; it is #774's parent, and the linkage to it is inherited, not declared. It is still stated, still linked, and now unmistakably an inherited relation rather than a second first-class one.

What I did not do, and why. I did not remove the reference to #774. It is the issue this pull request implements, and it is the one issue whose declaration is not inherited. Stripping it would leave the PR declaring a relationship only to the grandparent while the issue it was opened against goes unnamed — the inverse of the defect the rule exists to prevent. The form is Part of rather than Closes precisely because ## Not addressed records a partial delivery; a closing keyword is what that section rules out, not the reference itself. This is also the standing instruction for this run: keep the #774 reference, keep #775 as prose, add no other direct declaration, and use no closing keyword for an inherited issue. The body now satisfies all four.

Code, tests, docs

No code change this round — the review recorded no code findings, and confirmed the Round 1 mint-path concern resolved. Nothing user-observable changed, so no documentation, manual, or CHANGELOG.md entry applies.

Push and CI

Nothing to commit: the working tree is clean and the local branch is identical to origin/sehkone/issue-774 at 996e0894. A pull request body edit does not change the tree, and I did not manufacture an empty commit to force a re-run — the full suite is already green on that exact SHA, which is the SHA the reviewer read:

• Quality Check, Unit & CLI Smoke, Instructions / check, CodeQL, Analyze (rust/python/actions) — all pass
• All 13 Docker E2E matrix arms — all pass

The head SHA for this round is therefore unchanged at 996e08948148e586ee1e5ecc423fb2fdfc66d554.

[sehkone]

[Reviewer Round 3]

Changes requested.

• PR Probe the audit store's capacity and report it on the endpoint (#774) #965 body — the Round 2 rationale correctly preserves Part of #774: it is the implemented issue’s permitted partial-delivery reference. However, the body now mentions Reserved audit store: layout, enforced reserve, and capacity reporting #775 only in prose. The review contract specifically expects Part of #775 beside that reference; restore it as a direct declaration. The prior request to remove Part of #774 should not be carried forward.

The mint-path test concern remains resolved, and I found no additional code or issue-coverage defects in the updated diff.

[sehkone]

[Review Verdict Round 3: NOT_APPROVED]

[sehkone]

[Author Round 3]

Body edited on PR #965; nothing else touched.

Disposition

Conflicting-declaration item — Fixed (per operator resolution). The PR body's direct declarations are now exactly the two required ones, each on its own line at the top:

Part of #774
Part of #775

Part of #774 is this PR's permitted partial-delivery reference; Part of #775 is the inherited umbrella reference, now declared directly rather than carried only as prose. No closing keyword is used for either — a grep over the whole body confirms zero Closes/Fixes/Resolves occurrences.

I kept an explanatory sentence, reworded so it no longer contradicts the new lines (the old one asserted the #775 linkage was not declared directly, which is now false). It reads: "#774 is the issue this pull request delivers; #775 is the umbrella it hangs under, and the reference to it is inherited through #774 rather than being work this pull request implements directly."

Everything below that paragraph is byte-identical to the previous body (verified by diff — the only delta is GitHub's trailing newline).

No code change, no new CI run, as instructed. The working tree is unchanged, head stays 996e08948148e586ee1e5ecc423fb2fdfc66d554, and its existing 21 passing checks stand.

I understand this override applies to this run only and only to the direct-declaration point; the rest of the create-PR instruction remains in force.

COMPLETED

[sehkone]

[Reviewer Round 4]

[Reviewer Round 4]

Approved — no unresolved items or new findings.

The prior PR-linkage finding is resolved: the body now directly declares both Part of #774 and Part of #775, with no closing keyword. The updated diff continues to satisfy the issue’s capacity-probe, endpoint relay, failure-preservation, and documentation requirements.

[sehkone]

[Review Verdict Round 4: APPROVED]

[sehkone]

Suggested squash commit

Title

Probe the audit store's capacity and report it

Body

A reserved audit store with no measurement turns a slow fill into a
sudden outage on the host that must not be restarted, and a look-back
window that has quietly shrunk stays invisible. The store's own
signals stop at the bootroot host, so nothing carries them to an
operator's console.

An endpoint-enabled daemon now measures the store on the rotation
loop's existing maintenance tick and reports what it found in every
response's `registrar_health.audit_capacity` member — on refusals as
well as successes, because the store is a fail-closed control and a
success-only signal would stop carrying the alarm in exactly the
state the alarm exists to announce.

Both halves of the headroom come from one `O_NOFOLLOW` open of the
store root, so they describe one object rather than two resolutions of
the same path. Usage is measured per enforcement mode, since
`statvfs` describes a filesystem and not a subtree: `filesystem` mode
reads it from the same `fstatvfs`, while `directory` mode walks the
store by descriptor. The walk descends with `openat`/`fdopendir`
rather than by path because `openbao/` is written by the container's
uid by design, so a directory substituted under an active walk is an
attacker-reachable event rather than a hypothetical one.

`audit_store_low_water_bytes = 0` is now refused at load: at zero the
low-water band is empty, so the state machine would step straight from
exhausted to ok and the alarm this member exists to raise could never
fire.

Part of #774
Part of #775