flake: sigterm_shutdown server shuts down gracefully during startup, before readiness (exit 0, not a port conflict)

[TinDang97]

Summary

sigterm_clean_exit_shards_4_held_conn (tests/sigterm_shutdown.rs) intermittently fails
because the spawned server shuts down gracefully during startup, before the test's
readiness probe ever succeeds:

[s4-held] server process exited with ExitStatus(unix_wait_status(0))
          before becoming ready on port 49587

Observed once in scripts/ci-local.sh (macOS host tokio suite) under external CPU load.
6 of 7 tests in the binary passed.

What the server log shows

This is not a bind failure or a port conflict. The server came all the way up — 4 shards,
page caches, checkpoint managers, spill threads — and then ran the clean shutdown path
~60 ms after start:

08:29:23.024  Starting with 4 shards
08:29:23.056  Shard 0..3: PageCache initialized
08:29:23.062  Shard 0..3: checkpoint manager initialized
08:29:23.078  Shard 0: spill background thread initialized
08:29:23.078  Shard 0 shutting down          <-- ~60ms after start, before ready
08:29:23.082  Shard 1 shutting down
08:29:23.086  Shard 3 shutting down
08:29:23.090  Shard 2 shutting down
08:29:23.192  Server shut down

stderr carries only the benign <jemalloc>: option background_thread currently supports pthread only. No error, no panic.

Exit status is 0, which per this file's own doc comment means the graceful path ran to
completion — a signal death would report code() == None. So the process was asked to shut
down; it did not crash.

Why this is not the known port flake

The test already uses common::spawn_listening_guarded, the post-#284 helper with
respawn-on-dead-child, and the port is reserve_port-deduped. The failure mode here is a
server that binds and initialises successfully and is then told to stop. That is a different
class from #489 (unique_port()+N unverified ports → connection refused).

Mechanism: NOT determined

I have not established what requests the shutdown that early, and I do not want to guess.
Candidates worth checking, in no particular order:

• a SIGTERM intended for a previous child arriving after pid reuse
• process-group signalling from a sibling test in the same binary (7 tests, all of which
  spawn servers and signal them)
• the shutdown path being armed before readiness is published, so a signal delivered in the
  startup window is honoured rather than deferred

The third would be a genuine server-side ordering bug rather than a test bug, which is why
this is filed rather than waived.

Reproduction

Load-dependent. --test sigterm_shutdown in isolation: 0/6 failed (7 passed each run).
The single observed failure was during a full ci-local run with a foreground app consuming
~175% CPU.

Not caused by the branch it appeared on

Seen on docs/claude-md-native-scope, a docs-only branch whose
git diff main -- src/ tests/ benches/ Cargo.* is empty.

Related

• flake: manifest commit tests can observe another test's TEST_INJECT_PERSIST_ERROR (process-global knob, guard not taken by readers) #750 — separate manifest test-isolation race found in the same ci-local session
• crash_matrix_per_shard_aof: unique_port()+N uses unverified ports, causing 'Connection refused' flakes #489 — unique_port() unverified ports (different class, same general area)

[TinDang97]

Investigated. One of the stated premises was wrong, and it is now fixed — see #757.

This issue reasoned: exit status is 0, therefore the graceful path ran, therefore the process was asked to shut down. That does not follow. main() logged a fatal listener failure and then fell straight through to the normal shutdown sequence and Ok(()):

if let Err(e) = server::listener::run_sharded(...).await {
    tracing::error!("Listener error: {}", e);   // logged, not propagated
}
...
info!("Server shut down");
Ok(())                                          // <- exit 0, always

So a server that never bound its port exited 0, producing exactly the observable this issue recorded: shard-shutdown lines, Server shut down, exit 0, before readiness. Reproducible in one command:

$ moon --bind 203.0.113.7 --port 6399 --shards 1 --dir /tmp/x
ERROR moon: Listener error: Can't assign requested address (os error 49)
INFO  moon: Server shut down
$ echo $?
0

That is a production defect in its own right (systemd Restart=on-failure never restarts such a server), fixed in #757: the failure is returned after the shutdown sequence, so the exit status is non-zero while the AOF still flushes and the shards still join. A clean SIGTERM shutdown still exits 0, pinned by a test in the same file.

On the three candidates listed above:

• process-group / name-based signalling from a sibling test — checked, no. The only pkill in tests/ is -9 and scoped to its own dir marker; SIGKILL reports code() == None, not 0.
• pid reuse — send_sigterm targets a live child.id() immediately after a successful readiness probe, so reuse would need an intervening reap that does not happen.
• shutdown armed before readiness — structurally true and by design: SIGTERM is blocked at the top of main() before any thread spawns, and the handler thread sigwaits it, so a signal arriving during startup is queued and honoured rather than lost. But that only explains the shape given a sender; it does not supply one.

I have not proven what the observed occurrence was, and I am not closing this on a theory. What changed is that the evidence which ruled out a listener error was unsound, and the next occurrence will identify itself: exit 0 with Shutdown signal received means a real signal, non-zero with Listener error means a startup failure. The readiness failure message in tests/sigterm_shutdown.rs now names which line to look for.

Leaving this open until an occurrence is captured with the discriminating output.