Renew registrar leaves and honour the reload contract

[sehkone]

Renew registrar leaves and honour the reload contract

Context

The registrar surface uses two expiring certificate/key pairs: a registrar client leaf that the caller presents and an endpoint server leaf that callers pin. The daemon already performs start-time issuance before endpoint activation, has a profile-renewal loop with an immediate first pass, configured cadence and shutdown ownership, and exposes server-swap and per-dial client-load seams. It does not yet put these leaves on that lifecycle.

This issue adds one daemon-owned renewal adapter for both leaves, their in-process renewal state, safe server/client reloads, and the caller reload documentation. The open reporting sibling owns wire-visible health reporting and typed lapse classification; this issue supplies its in-process input but puts nothing on the wire.

Scope

Create the registrar-certificate renewal adapter in the daemon lifecycle. It is created only when the endpoint is enabled, owns both leaves, receives the daemon shutdown signal, and its retained task handle joins with the daemon's other tasks. It runs immediately, then uses the rendered internal credential agent configuration's sole profile: its daemon check interval and jitter set cadence, renew_before is lead time, and its retry configuration governs issuance retries.

Initialize the shared per-leaf renewal-state accessor after the existing start-time issuance has ensured usable material and before endpoint serving begins. Read each enabled leaf's valid certificate and record notAfter, never_attempted, and no attempt time. This is not an attempt and does not contact OpenBao or the CA.

On each tick, renew a leaf inside lead time or whose chain no longer verifies against trust.ca_bundle_path; lead time is checked first. The shared eligibility predicate retains its no-bundle rule: with ca_bundle_path unconfigured, chain drift does not trigger renewal. That state cannot activate an enabled endpoint: build_server_config requires both trust.ca_bundle_path and non-empty [trust].trusted_ca_sha256, with no system-root or pin-file fallback for incoming mTLS. Preserve and test the predicate below activation; do not create a verifier fallback. Use the existing daemon eligibility and issuance-retry policy.

Refactor registrar issuance for renewal so it returns the chain, candidate certificate, and fresh key without writing live output—not the configured pair and not trust.ca_bundle_path. Candidate files use the finished files' required permissions; a private rollback snapshot is written no wider than the key mode whatever the file it copies, and records that file's own mode so a restore puts it back. All of them are removed on every exit after creation, including issuance, validation, publication, rollback, and TLS-swap failure. Validate every candidate before a live write: its key matches the leaf; a client candidate has the reserved registrar SAN and recognises to the same instance/host/domain; a server candidate has the exact endpoint SAN and chains to an anchor in the existing endpoint pin file. Start-time issuance may use the new helper, but renewal must not call the present direct-to-live issue_surface_pair path.

The endpoint pin file and the incoming client verifier have distinct roles. The pin file decides only whether a replacement server leaf is safe for pinned callers; it never supplies incoming mTLS roots. Rebuild WebPkiClientVerifier from the would-be post-merge bytes of trust.ca_bundle_path, retaining exactly PEM certificates whose DER SHA-256 is in [trust].trusted_ca_sha256. Do not use the returned chain alone, the endpoint pin file, all bundle entries, or system roots. Before live publication, build a complete next ServerConfig from staged inputs: the candidate server pair when renewing that leaf, the current live server pair when renewing only the client leaf, and the staged merged bundle in either case. This build must finish before any live path changes.

Publish a validated renewal as a recoverable transaction. Before the first live write, capture restore artifacts for the affected live bundle and pair, including bytes, final modes, and ownership. Atomically replace the merged bundle, write the live certificate then key through the established two-rename contract, and only then atomically exchange the already-built active TlsAcceptor/configuration. The accept loop loads the active acceptor immediately before each handshake and holds no lock across an await. Replacing EndpointCertResolver alone is insufficient because the old acceptor retains the old client verifier. Existing handshakes and connections keep their old configuration and are not dropped; the socket is not rebound.

The active configuration changes only after every live file write succeeds; its exchange is infallible because the full replacement was built first. If a bundle, certificate, or key write fails, do not exchange it; restore every live path whose publication may have started from the saved artifacts, leave the old active configuration installed, record a failed attempt with the publication and, if applicable, rollback error, and retry on the next tick. A successful rollback restores prior bytes, modes, and ownership. If rollback itself fails, do not conceal the mixed disk state or claim it was restored: keep the old active configuration, record both errors, clean all private artifacts, and let a later tick retry. The per-dial loader remains responsible for never presenting a mismatched pair. An invalid or unpinned server candidate is a refusal: discard it, record the pin-file reason, and change no live or active state.

Each successful issuance uses a fresh key. Never rewrite the endpoint pin file or add a leaf fingerprint. During CA rotation the tick chooses no ordering and does not repair the bootroot-internal credential. The rotation's mandatory Phase-4 tail repairs that credential under additive transitional trust before Phase 5 reissues services. If the tick reaches a registrar leaf while the credential cannot authenticate, record an ordinary failed issuance and retry next tick. On chain drift the client leaf is reissued; reissue the server leaf only when its candidate anchor is already in the unchanged pin file.

Update the accessor only when initialization or an attempt supplies state. A success replaces notAfter and records succeeded with attempt time. An issuance, publication, rollback, authentication, or pin-file-refusal failure retains notAfter and records failed with attempt time and reason. A no-op tick changes nothing. Enabled endpoints have entries for both leaves.

Honour the client reload contract through the completed in-repository per-dial loader. Each dial rereads its pair and verifies key-to-leaf matching. The writer uses two renames, so a reader in the interval retries rather than presenting a mismatch: five reads total, waits of 1 ms, 2 ms, 4 ms, and 8 ms after the first four mismatches, then a typed mismatch error. This fixed local policy is not the daemon issuance-retry policy. Document this contract in the existing English and Korean endpoint documentation, naming the in-repository client as reference behaviour.

Acceptance criteria

• With the endpoint enabled, one adapter registers both leaves, initializes both accessor entries from valid on-disk certificates before serving, and joins on shutdown. With it disabled, no adapter, state entries, renewal work, OpenBao request, or CA request exists.
• The first pass is immediate; later passes use the internal agent configuration's sole profile cadence, jitter, lead time, and issuance retry settings. Tests can drive a pass without wall-clock waiting.
• Initialization records observed notAfter, never_attempted, and no timestamp without an issuance request; one driven pass cannot read role_id/secret_id or use AppRole.
• A due leaf is issued off-live with a fresh key. Candidate key/SAN/pin validation, staged CA-bundle construction, next TLS configuration construction, and restore snapshots precede every live write; candidates and snapshots are cleaned on every exit.
• A client candidate preserves its recognised instance, host, and domain. A server candidate has the exact endpoint SAN and chains to an existing endpoint-pin-file anchor.
• With a configured bundle, an in-date leaf outside lead time whose chain no longer reaches it is eligible through the shared drift check; without it, drift does not trigger renewal and the enabled endpoint refuses activation rather than falling back to other roots.
• For an unpinned server candidate, the live bundle, pair, active TLS configuration, and pin file remain byte-for-byte unchanged; the accessor records failed with the pin-file reason and attempt time.
• The rebuilt incoming verifier uses only staged post-merge trust.ca_bundle_path certificates whose DER fingerprints occur in trusted_ca_sha256; pin-file anchors, unpinned bundle certificates, a returned chain alone, and system roots cannot substitute.
• Successful publication replaces the merged bundle, writes certificate then key, and only then swaps the prebuilt active TLS configuration. The next handshake presents the renewed server leaf and accepts the renewed client leaf without restart, signal, socket rebind, socket-path replacement, or dropped in-flight connections.
• An injected bundle, certificate, or key publication failure retains the old active configuration, restores every changed live path from saved bytes/mode/ownership, records failure, and cleans private artifacts. An injected rollback failure retains the old active configuration, records both errors, cleans artifacts, and does not claim the live files were restored.
• A successful attempt updates notAfter and succeeded; every ordinary issuance, publication, rollback, authentication, or pin-file refusal failure retains notAfter and records failed; a no-op pass is unchanged.
• A reader forced between the two renames retries until it finds a matching pair and never presents a mismatch. A permanent mismatch performs exactly five reads, follows the 1/2/4/8 ms waits, and returns the typed mismatch error.
• No response field, serialization fixture, wire-visible renewal state, or typed lapse classification is added. English and Korean endpoint documentation state the per-dial reload contract.
• Changed Rust code passes the repository-required formatting and warning-free clippy invocations.

Constraints

• Do not create a second daemon, supervisor, scheduler, lead-time policy, or daemon issuance-retry policy.
• Do not use AppRole, role_id, secret_id, or the per-service ACME renewal path.
• Do not publish renewal output before candidate key/SAN/server-anchor, staged-bundle, and next-TLS-configuration validation completes.
• Do not make the pin file, returned chain alone, all bundle entries, or system roots an incoming client-verifier source.
• Do not rewrite the endpoint pin file, add a leaf pin, swap active TLS after a publication failure, or describe a failed rollback as restored.
• Do not restart the daemon, rebind or unlink its socket, require an external signal, or drop in-flight connections for a TLS replacement.
• Do not repair certificates at start, repair the internal credential, add request-path recovery, or impose rotation ordering from this tick.
• Keep tests deterministic: inject/control ticks, publication failures, rollback failures, and loader delays.

Out of scope

• Initial issuance, start-up repair, certificate naming, and CSR construction.
• The endpoint pin-file format and its provisioning tool.
• Renewal or recovery of the bootroot-internal privileged credential.
• Health-response schema, lifetime reporting, lapse precedence, and caller-visible lapse errors.

Test plan

• Test enabled/disabled adapter creation, initialization, shutdown, and absence of disabled-path OpenBao/CA activity.
• Test immediate and scheduled ticks with controllable time and shutdown; assert no-AppRole access.
• Drive lead-time renewal and assert candidate validation, staged verifier build, and snapshot creation before publication; assert fresh key, preserved client identity, valid server SAN, unchanged pins, and accessor success.
• Drive drift: verify client replacement, server publication with a pinned incoming anchor, and unpinned-server refusal with unchanged live/active state and retained notAfter. Verify no-bundle drift opt-out and endpoint activation refusal.
• Test the incoming verifier's exact source set, including rejection of pin-file-only, unpinned-bundle, returned-chain-only, and no-bundle substitutes.
• Test successful active configuration replacement and mTLS after trust-anchor rotation; assert unchanged socket inode and undropped in-flight connection.
• Inject each live publication failure and assert rollback, old acceptor, failed state, and cleanup. Inject rollback failure and assert old acceptor, both errors, and cleanup.
• Test an ordinary issuance/internal-credential authentication failure and a no-op pass.
• Test next-dial client renewal without caller recreation and the deterministic torn-pair success/exhaustion paths.
• Run documentation build and required formatting, lint, unit, and applicable Docker E2E preflight checks.

Dependencies

Part of #770. The callable registrar issuance unit, endpoint certificate-swap seam, and in-repository per-dial client loader are already present in origin/main.

Pointers

• src/bin/bootroot-agent.rs for start-time issuance and activation order.
• src/daemon.rs for task ownership, cadence, shutdown, and eligibility.
• src/registrar_certs.rs (issue_surface_pair) for the publication boundary.
• src/acme/flow.rs (issue_certificate_with_bootstrap) and src/fs_util.rs (write_cert_and_key) for current bundle/pair publication.
• src/registrar/internal/agent_config.rs for renewal settings.
• src/cert_chain.rs for chain-drift verification.
• src/registrar/endpoint.rs, src/registrar/endpoint/serve.rs, and src/registrar/endpoint/tls.rs for active acceptor replacement and verifier construction; src/registrar/endpoint/client.rs for the loader.
• src/registrar/endpoint_pin.rs for server-anchor validation.
• docs/en/operations.md and docs/ko/operations.md for caller documentation.

[sehkone]

Major discrepancy: the implementation does not deliver the issue's required deterministic test plan for the renewal transaction and daemon lifecycle. The added tests cover state initialization, a no-op pass, an injected issuance failure, standalone snapshot restoration, one expired client candidate, and torn-pair retries. They do not drive the enabled/disabled adapter lifecycle or controlled scheduled ticks; real off-live issuance and publication; drift and unpinned-server refusal; exact staged-verifier trust sources; mTLS replacement after trust rotation; or injected bundle/certificate/key publication and rollback failures. These are explicit acceptance and test-plan requirements, so this needs implementation/test work rather than an issue-body clarification.