Summary
do_rewrite_per_shard (src/persistence/aof.rs ~2880-2972) runs synchronously on the dedicated per-shard writer thread. If it panics (e.g. save_snapshot_to_bytes OOM-unwinds), the thread unwinds WITHOUT calling coord.shard_done(). Consequences:
- The
AtomicUsize rewrite countdown never reaches zero → no shard commits the manifest (old generation stays intact — crash-safe, good).
AOF_REWRITE_IN_PROGRESS (set in bgrewriteaof_start_sharded, src/command/persistence.rs ~308) is never cleared → all future BGREWRITEAOF calls are permanently blocked.- The writer thread for that shard is dead → subsequent
AofMessage::Append sends fill the bounded (10k) queue and are silently dropped → that shard runs with NO AOF durability for the rest of the server's lifetime.
Consequence (3) is the real impact: silent per-shard durability loss, not just a stuck flag.
Severity
LOW (gated behind --experimental-per-shard-rewrite; only triggers on a panic during the fold). Pre-existing; surfaced in PR #136 self-review.
Suggested fix
Wrap the fold in std::panic::catch_unwind, or use a RAII guard that calls coord.mark_failed(); coord.shard_done(); on drop and clears AOF_REWRITE_IN_PROGRESS; add a writer-thread restart (or at minimum a liveness metric) on panic.
Context
Found during PR #136 review. Reviewer-reported, not independently reproduced.
Summary
do_rewrite_per_shard(src/persistence/aof.rs ~2880-2972) runs synchronously on the dedicated per-shard writer thread. If it panics (e.g.save_snapshot_to_bytesOOM-unwinds), the thread unwinds WITHOUT callingcoord.shard_done(). Consequences:AtomicUsizerewrite countdown never reaches zero → no shard commits the manifest (old generation stays intact — crash-safe, good).AOF_REWRITE_IN_PROGRESS(set inbgrewriteaof_start_sharded, src/command/persistence.rs ~308) is never cleared → all futureBGREWRITEAOFcalls are permanently blocked.AofMessage::Appendsends fill the bounded (10k) queue and are silently dropped → that shard runs with NO AOF durability for the rest of the server's lifetime.Consequence (3) is the real impact: silent per-shard durability loss, not just a stuck flag.
Severity
LOW (gated behind
--experimental-per-shard-rewrite; only triggers on a panic during the fold). Pre-existing; surfaced in PR #136 self-review.Suggested fix
Wrap the fold in
std::panic::catch_unwind, or use a RAII guard that callscoord.mark_failed(); coord.shard_done();on drop and clearsAOF_REWRITE_IN_PROGRESS; add a writer-thread restart (or at minimum a liveness metric) on panic.Context
Found during PR #136 review. Reviewer-reported, not independently reproduced.