[Security][tentative] Medium (attackable): Startup Backfill Can Be Dropped Before Scheme Registration

[evonide]

Context

Node startup wires the finalizer, syncer, and orchestrator together. The orchestrator registers epoch-specific consensus verification schemes in SummitSchemeProvider, while the syncer handles repair and resolver-delivered backfill that must be verified before it is stored or acted on.

Current-epoch finalized or notarized data should either be verifiable when the syncer accepts it, or remain retryable until the verifier is available. That invariant depends on startup ordering between syncer repair/backfill and current-epoch scheme registration.

Claim

A malicious sync peer that controls responses to a restarting node's repair/backfill requests can trigger this only if its current-epoch finalized or notarized response is delivered in the scheduler window after syncer delivery handling is live but before the orchestrator processes the queued epoch-entry message. In that window, syncer acknowledges the delivery as handled without verification or storage and can leave the node's recovery gap unfilled. Engine startup has no explicit barrier that prevents syncer repair/backfill deliveries from being handled before the orchestrator has registered the current-epoch verifier. If that scheduler window is hit, missing-scheme deliveries are treated as handled instead of retryable, so valid startup backfill can be dropped without verification or storage.

Flow

The bug branch is reachable once a delivery is presented while SummitSchemeProvider has no scheme for the delivery epoch. The start condition is narrower than plain restart: engine start order starts finalizer and syncer before orchestrator, but those actors are spawned asynchronously, so source review proves an unbarriered race rather than a deterministic startup ordering where repair/backfill must always beat scheme registration.

Impact

A restarting or catching-up node can receive the missing finalized certificate or notarized block it needs during startup, acknowledge it as successfully handled, and then fail to store or process it because the verifier was not registered yet. Recovery can stall until another event happens to re-request the same data.

Root Cause

Startup ordering permits syncer delivery handling before orchestrator scheme registration, and missing-scheme deliveries return success instead of a retryable failure.

Code

• Engine startup starts finalizer, then syncer, then orchestrator: https://github.com/SeismicSystems/summit/blob/ed2c5c8/node/src/engine.rs#L453, https://github.com/SeismicSystems/summit/blob/ed2c5c8/node/src/engine.rs#L455, https://github.com/SeismicSystems/summit/blob/ed2c5c8/node/src/engine.rs#L462.
• Finalizer queues the epoch entry message to the orchestrator: https://github.com/SeismicSystems/summit/blob/ed2c5c8/finalizer/src/actor.rs#L253.
• Orchestrator registers the scheme only when it later handles that message: https://github.com/SeismicSystems/summit/blob/ed2c5c8/orchestrator/src/actor.rs#L235, https://github.com/SeismicSystems/summit/blob/ed2c5c8/orchestrator/src/actor.rs#L238.
• SummitSchemeProvider starts with an empty scheme map and scoped returns only registered schemes: https://github.com/SeismicSystems/summit/blob/ed2c5c8/types/src/scheme.rs#L49, https://github.com/SeismicSystems/summit/blob/ed2c5c8/types/src/scheme.rs#L97.
• Syncer runs startup repair and dispatch before normal loop processing, but actor spawning means this is not evidence that syncer startup always races ahead of orchestrator registration on every restart: https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L465, https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L473.
• If finalized delivery lacks a scheme, syncer logs it as stale and acknowledges success: https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L928, https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L934.
• Notarized delivery has the same no-scheme success acknowledgement: https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L966, https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/actor.rs#L972.
• The resolver consumer treats that boolean as successful delivery: https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/ingress/handler.rs#L63, https://github.com/SeismicSystems/summit/blob/ed2c5c8/syncer/src/ingress/handler.rs#L78.

Related Issues/PRs

The following existing items touch the same trust boundary, adjacent root cause, or likely remediation stack.

• [Security][tentative] Medium (correctness): Old Epoch Schemes Are Unregistered Too Early #251
• [Security][tentative] Medium (correctness): Backfill Resolver Uses A Hard-Coded Short Fetch Timeout #306
• [Security][tentative] Medium (attackable): Finalized Sync Does Not Enforce Parent Linkage #324
• [Security][tentative] Medium (correctness): Repaired Finalized Ancestors Are Archived Without Certificates #344
• [Security][tentative] Medium (attackable): Resolver-Cancelled Subscriptions Stay Pending #345
• [Security][tentative] Medium (correctness): Syncer SetFloor Stale Round Fetches Below Floor #357

Note that #296 is the startup ordering window before the epoch verification scheme is registered; the others cover old-epoch schemes, finalized backfill, and resolver/subscription recovery behavior.

Fix

• Start/register the orchestrator scheme before allowing syncer repair/delivery processing.
• Treat missing current-epoch schemes as temporary failure and return false so resolver retries.
• Add a startup barrier between finalizer epoch entry and syncer backfill.

[matthias-wright]

This issue was inherited from upstream marshal and still exists on the main branch.
There are two reasons why I think we don't have to address this:

1. This race can only happen on the backfill path and not during live consensus.
2. It is self-healing. Once the next finalization arrives, try_repair_gaps will be triggered and the missing blocks/certs will be requested again.

[sebastian-osec]

That makes sense for severity: this is backfill/recovery-only and should not affect live consensus.

The remaining question is whether the self-healing path is guaranteed after the early delivery has already been acknowledged as successful. If a later finalization or repair pass always re-triggers try_repair_gaps for the missing data, then I agree this can be treated as a bounded recovery race rather than an actionable issue. If there is any startup state where the success ack suppresses the only outstanding fetch and no later repair signal is guaranteed, then the residual recovery concern remains.

[matthias-wright]

I looked into this again. The finalization path is indeed self-healing via try_repair_gaps. I think we can close this issue.