[Umbrella] Make the audit store's reserve a kernel-enforced ceiling

[sehkone]

Make the audit store's reserve a kernel-enforced ceiling

Summary

The registrar's two audit artifacts — the daemon's verb records and the OpenBao file audit device — already share one directory on the bootroot host, sized by [registrar] audit_store_reserve_bytes and placed by audit_store_dir. What that store does not have is a ceiling. The reserve is a number written down beside a location and nothing enforces it, which the code says in as many words today: the filesystem enforcement variant is documented as future work and both manuals carry the same disclaimer.

A configured number over a plain directory is a monitoring control wearing a reservation's name. The artifacts stay able to fill the host's root filesystem, and on this host that is not a lost log — the OpenBao file audit device is mandatory, OpenBao fails requests it cannot audit, and the host runs a shamir-sealed OpenBao whose unplanned restart costs a manual unseal. An unbounded audit artifact can stop certificate renewal for the whole deployment.

This effort makes the reserve real without repartitioning a host or assuming a quota-capable filesystem: a fully allocated loopback image sized to the reserve, an ext4 filesystem on it, and a systemd mount unit that mounts it at audit_store_dir and is restored on boot. The ceiling is then the image size — a quota by construction, enforced by the kernel against both writers, with ENOSPC arriving at the reserve instead of at the host's root filesystem.

It lands as four issues, and they are not one chain. Provisioning comes first alone, because a mount hides whatever is underneath it: it builds and verifies the ceiling on the fresh-host path, fixes what enforced means, and on a store that already holds records refuses to activate rather than mounting over them. Two issues then hang off it and off nothing else, so they can be worked in parallel — one puts the same pass in front of the command a live deployment can actually re-run and stops Docker manufacturing the audit bind source at boot, the other stops the daemon serving the registrar verbs while the store is not mounted. The fourth is the operator procedure that moves an existing store's records onto the reserve without losing one and without leaving a half-finished state that reads as success. It exists to resolve the refusal provisioning raises, and it carries the one other operation that moves the same records — replacing the image when audit_store_reserve_bytes changes — under the same copy and verification contract rather than a second and thinner one.

Children

• Provision and verify the audit store's loopback-backed reserve #923 Provision and verify the audit store's loopback-backed reserve
• Refuse the bring-up and the boot path on an unmounted audit store #924 Refuse the bring-up and the boot path on an unmounted audit store
• Refuse the registrar verbs while the audit store is not mounted #925 Refuse the registrar verbs while the audit store is not mounted
• Move audit records onto the reserve, at activation and when the reserve changes #926 Move audit records onto the reserve, at activation and when the reserve changes

The four issues form a diamond, not a chain. Provisioning is the single root: it derives the image and the mount unit, renders the two ordering drop-ins, verifies the whole chain end to end, defines the four reported outcomes, and refuses to activate over a store that already holds records. Everything else is written against the outcomes it fixes, so nothing can precede it.

Two issues then depend on provisioning and on nothing else, and are deliberately parallel. The installer half extends that same pass to bootroot infra up — the only one of these commands an operator on a live deployment can re-run, since bootroot init bails on an already-initialised host — and declares the audit bind with create_host_path: false so a mount job that failed at boot cannot end with Docker manufacturing the bind source on the root filesystem. The daemon half refuses the registrar verbs while the store is not mounted, so a started daemon cannot manufacture its record directory beneath an empty mount point; it reaches that with a mount-point check ahead of the handler build rather than by touching the record store's own directory creation. The two share no code and may land in either order. What they do share is one passage of each manual and one message catalogue, so whichever lands second extends what the first wrote rather than opening a second section.

The relocation issue closes the diamond, and depends on the installer half rather than on both. It carries every procedure that moves an audit store's contents under one copy-and-verification contract: the relocation that resolves the refusal provisioning raises, and the numbered steps that replace the image when audit_store_reserve_bytes changes. It waits on the installer half for two reasons — bootroot infra up is the closing verification each of those procedures ends with, and the residual it finally retires is one the bind guard opens rather than closes, since a store still carrying an openbao/ directory underneath gives Docker a path to bind and the guard cannot tell that apart from a mounted one. It does not wait on the daemon half: the relocation holds the daemon out by stopping its service, which is the only mechanism that survives the mount coming up, so an edge there would be false and would lengthen the critical path for nothing.

The schedule risk sits with the three that carry code: between them they touch the installer, generated systemd artifacts, the rendered Compose override and its reader, the daemon's endpoint-handler build, and the E2E lifecycle scripts, and their acceptance rests on assertions that need a real loop device. The relocation issue is almost entirely rendered commands and metadata reads, so it carries little build risk — but it is the one holding the audit trail in its hands, and its verification gate is where review attention belongs. Two rules cut across the three that carry code and are easy to leave implicit. Every byte figure derived from a statvfs or a stat — available space, allocated size, a subtree sum, a size plus a margin — is checked arithmetic whose failure is a refusal, because a product that wraps in a release build turns "does not fit" into a pass on exactly the check that protects the records. And two things the provisioning child builds are called by the relocation child rather than rebuilt: the walk that sums a subtree's allocated bytes, whose hard-link and device-boundary rules must give one answer across the effort, and the renderer that produces the outstanding activation commands, whose steps the relocation child re-emits with exactly one withheld — so the provisioning child owes both a shape a later caller can use.

Execution order

Issues in the same wave have no unmet dependencies among these children and can run in parallel.

• Wave 1: Provision and verify the audit store's loopback-backed reserve #923
• Wave 2: Refuse the bring-up and the boot path on an unmounted audit store #924, Refuse the registrar verbs while the audit store is not mounted #925
• Wave 3: Move audit records onto the reserve, at activation and when the reserve changes #926

Edges, stated so the diamond is readable without the prose above:

• Refuse the bring-up and the boot path on an unmounted audit store #924 depends on Provision and verify the audit store's loopback-backed reserve #923.
• Refuse the registrar verbs while the audit store is not mounted #925 depends on Provision and verify the audit store's loopback-backed reserve #923.
• Move audit records onto the reserve, at activation and when the reserve changes #926 depends on Refuse the bring-up and the boot path on an unmounted audit store #924, and through it on Provision and verify the audit store's loopback-backed reserve #923. It does not depend on Refuse the registrar verbs while the audit store is not mounted #925: the relocation holds the daemon out by stopping its service, so an edge there would be false and would lengthen the critical path for nothing.

External dependencies: none. #923 reads only what is already on the default branch.

graph TD
  issue923["#923 Provision and verify the reserve"]
  issue924["#924 Refuse bring-up and boot on an unmounted store"]
  issue925["#925 Refuse the registrar verbs while unmounted"]
  issue926["#926 Move records onto the reserve"]
  issue923 --> issue924
  issue923 --> issue925
  issue924 --> issue926

Loading

Shared background

What enforcement does and does not buy, stated up front so no child's documentation overclaims. Isolation bounds the blast radius to the audit artifacts and protects everything else on the host — that is real, and it is what a configured number does not deliver. What it cannot deliver is that exhausting the artifacts "cannot stop OpenBao serving": OpenBao fails requests when its own device is unwritable, so a full audit filesystem stops it whether that filesystem is isolated or not, only sooner and with the rest of the host intact. Bounding the OpenBao device's own growth is separate work.

The host surgery is the operator's, and the reason has to be stated accurately. Every one of these runs is already root — bootroot init refuses to provision the store when its effective uid is not the production uid, before it creates anything — so privilege is not the obstacle, and no child may describe its own half as "the unprivileged pass" on that basis. What the children do is render, read and report; what the operator does is every step that changes the host. Two other things are. mkfs.ext4 and mount need host tooling and a loop device bootroot has never required and cannot assume. And the choices they make — which device backs the image, how it composes with existing quotas, what the host's mount policy allows — are site-specific and destructive to guess on the one host this design says must not be restarted. Every other systemd artifact in this repository reaches a host by being installed by an operator, so every child follows that route: bootroot renders exact commands, the operator runs them, and bootroot verifies and fails closed until verification passes.

Where this attaches: three commands, and not bootroot infra install. bootroot init provisions the audit store before any Docker call, from the [registrar] keys it reads through --agent-config, and it is where the provisioning phases, the rendered privileged commands and the reported outcome live. bootroot reinit raises the same refusals ahead of its destructive wipe. bootroot infra up re-checks the store before the stack comes up, and it is the closing verification of every maintenance procedure this effort documents — because bootroot init bails on an already-initialised host, so it is not a command an operator on a live deployment can re-run. bootroot infra install installs and starts the Compose stack, runs ahead of bootroot init on a fresh host, and has neither --agent-config nor any read of [registrar]; it is unchanged by this work.

Two writers, and which of them can exhaust the reserve is a fact about the configuration, not about the design. The verb records are self-bounding at audit_max_file_bytes × (audit_max_retained_files + 1) — about 136 MiB on the shipped defaults of 8 MiB and 16 retained files — while the OpenBao audit device, which bootroot does not bound and OpenBao does not rotate, has no bound at all. On the shipped 2 GiB reserve the device is therefore the writer that fills the store; raise either record key far enough, or lower the reserve far enough, and the records fill it too. filesystem mode refuses at provisioning time any reserve that does not strictly exceed the record figure, which catches the configurations that are provably too small. It does not establish that one which clears it is big enough: an ext4 filesystem offers less usable space than the image holding it, and the device has no bound to size against, so sizing the store so both writers fit stays the operator's arithmetic in either mode. Both manuals should carry the formula, the defaults and that condition rather than the conclusion alone, so an operator editing a retention key can see what it costs.

The boot path needs a refusal of its own, and the installer child carries it. Boot ordering cannot be made to guarantee the mount: a hard relation on docker.service would take the whole Compose stack down over a fault local to one directory, so the drop-ins are Wants= plus After= and nothing stronger. A mount job that fails at boot therefore does not stop Docker, restart: always brings the OpenBao container back, and Docker manufactures the missing bind source under the empty mount point — the mandatory audit device written to the root filesystem, which is exactly what an activated reserve is supposed to prevent, on the one path where neither an operator nor a bootroot command is watching. The only mechanism that closes it is declaring that bind with create_host_path: false in the rendered Compose override, so the container fails to start instead. That guard belongs with the bootroot infra up refusals rather than with the daemon's, being a property of the rendered override; it forces the long mount syntax on that one entry and a reader that accepts both forms, and bootroot infra up upgrades an override written before it. It is also narrower than it reads: it stops the container only where the bind source is genuinely absent, so a host whose underlying store still carries the openbao/ directory the existing layout created gives Docker a path to find, and the device lands on the root filesystem again. There the exposure survives until the relocation child renames the underlying store aside. So the provisioning child documents this residual at that full width, rather than implying that the drop-ins — or the bind guard alone — make the boot path safe.

Measuring the store belongs elsewhere. No child of this umbrella computes headroom, raises a capacity alarm or touches the endpoint's health response; a probe that reports the store's usage and available space is separate work that depends on the mode this effort makes real.

[sehkone]

Wiring added for the hand-completed filing

The refine run did not reach the end, so the four children were filed by hand. Their content is sound — the diamond is justified, and the two artefacts #923 owes #926 (the allocated-size subtree walk and the phase-1 renderer that emits the outstanding activation commands) are stated in both the producer and the consumer, so neither is left only in this roster. What the hand-filing could not carry was the machine-readable half, which is added here without touching any of the reasoning:

• This issue is retitled [Umbrella], matching the other umbrellas in this tree.
• ## Children now carries a numbered roster — Provision and verify the audit store's loopback-backed reserve #923, Refuse the bring-up and the boot path on an unmounted audit store #924, Refuse the registrar verbs while the audit store is not mounted #925, Move audit records onto the reserve, at activation and when the reserve changes #926 — ahead of the prose that explains the shape.
• ## Execution order is new, with the waves, the edges spelled out one by one, and a mermaid graph. That block is the authority for order: a children checkbox is prose written at one moment and is not a status.
• Each child's ## Dependencies now names its siblings by number where it previously described them ("the issue that provisions and verifies the reserve" → "Provision and verify the audit store's loopback-backed reserve #923, the issue that provisions…"). The reasoning is unchanged; only anchors were inserted.

The edges, for the record: #923 → {#924, #925} → #926 ← #924 only. #926 does not wait on #925, because the relocation holds the daemon out by stopping its service in step 1 — an edge there would be false and would lengthen the critical path for nothing.

One thing not fixed, recorded rather than papered over: this issue's AgentCoop node marker still reads leaf-audit-store-reserve, and the four children carry no node markers at all, because they were not produced by the run that would have stamped them. Nothing in the issue text depends on those markers; a later AgentCoop run over this tree may not recognise the children as its own nodes.