Skip to content

polis-api: stops cleanly on SIGTERM, keeps or counts every shadow observation in flight, and answers a request head it cannot parse - #3455

Open
colinmegill wants to merge 7 commits into
edgefrom
polis-api/sigterm-drain-and-bad-heads
Open

colinmegill wants to merge 7 commits into
edgefrom
polis-api/sigterm-drain-and-bad-heads

Conversation

@colinmegill

Copy link
Copy Markdown
Member

polis-api: stops cleanly on SIGTERM, keeps or counts every shadow observation in flight, and answers a request head it cannot parse

Changes to the Rust API process (queue-rs/polis-api) that the vote writer (POST /api/v3/votes) needs before it can ever serve participants, and to the comparison tool that reads the shadow's lines (queue-rs/src/bin/polis-shadow). Today the vote route is not routed to Rust and the live vote writer is off; production runs this binary as the pca2 server (polis-api), as the vote and comment shadow (polis-api-shadow) and as the read shadow (polis-api-shadow-reads). No request a participant sends is answered differently.

Before the live vote writer is ever switched on: an open condition

This change does not make the live vote writer safe to switch on, and does not claim to. With the writer on (POLIS_API_POST_VOTES=1 and postVotes routed), a stop of polis-api while the box's nginx still sends it votes has two outcomes that must be resolved, or explicitly accepted, before any live flip:

  1. A vote recorded twice. A vote dispatched before the stop is answered by its own deadline, one second after dispatch: 200 with its receipt once the commit is acknowledged, otherwise 202 {"acceptance":"unknown","retry":"same_vote"}. Both participation clients retry a 202 once, 300 ms later, with the same body. During the stop that retry finds polis-api's listener closed (nginx sees zero bytes sent) or, if the kernel had already queued its connection, gets the marked hand-back; either way nginx gives it to Node, and Node appends a history row on every vote request. If the original also commits (during the drain, or by a COMMIT processed just before the exit), the vote has two votes history rows of the same choice and one votes_latest_unique row. This is the case the vote contract already names as an activation condition (a retry routed to Node appends again); a stop is one more way into it. Before this change the process ignored SIGTERM and kept accepting, so such a retry reached the writer, which does not append a same-choice replay (or, while the original still held its row, answered the retry 500 after its 100 ms lock wait).
  2. A vote nothing stored. A vote answered 202 whose transaction is still running when the stop's bound ends (5 s of drain, then 0.5 s for blocking work) is cut when the process exits: nothing is stored if PostgreSQL had not processed its COMMIT. If its one retry was also cut (it reached the writer before the stop and waited behind the same congestion), nothing is stored and the client reports the vote as failed. The participant is never told such a vote was stored: a 200 is given only after the commit is acknowledged. Before this change the cut came at SIGKILL, 10 s after the signal, so this change can roll back a 202 vote that the old process would have let commit between 5.5 s and 10 s.

When the route into Node opens: only when this box's nginx still accepts requests while polis-api's listener is closed, for example a stop or recreate of the polis-api service alone. In the web deploy's own teardown nginx-proxy is stopped in the same --profile rust-api down, and with the deploy's load-balancer traffic control a box receives no new requests while its containers stop.

What would close it, each an addition to this change rather than a reversal of it: a stop window long enough for a dispatched vote's worst case (writer acquisition 5 s, connection setup 5 s, then its statements: a stop_grace_period on polis-api and a drain bound above that, about 13 to 15 s), and keeping a 202's retry off Node's append (a same-choice retry that Node does not append, or a retry routed back to the writer). None of this can happen while the writer is off.

What anyone would observe

Every deploy's stop of the three Rust containers

The web deploy's teardown (scripts/after_install.sh, --profile rust-api down, with no -t), docker stop and docker compose down send SIGTERM and, 10 s later, SIGKILL: Compose's default stop_grace_period, which none of the three services sets. Before this change each of them, as PID 1 in its container, ignored SIGTERM, ran the whole 10 s and was killed (exit 137).

  • The pca2 server (polis-api, main role). It now stops accepting connections at SIGTERM, serves the connections the kernel had already completed, finishes the requests it is answering, and exits 0: in about 0.15 s when idle, at most 5.5 s. A pca2 request after that finds the listener closed and nginx's existing fallback answers it from Node (the same bytes), where before it reached a process that was about to be killed. New lines: polis_api_stopping (which signal) and polis_api_stopped (whether it drained, how long it took, what it left, how many writes it handed back).
  • The vote and comment shadow (polis-api-shadow). It now stops accepting copies at SIGTERM and waits at most 7.5 s for the observations it has begun to print their lines. A copy's line follows its read-back of what Node stored, 2 s after the copy arrived, or 10 s after when Node's row is not there at the first look. So:
    • every copy whose row Node has stored by the first look gets its own line as before, and the stop then ends about 1.6 s after the signal, exit 0;
    • a copy still in flight at 7.5 s (in practice: waiting for its second look, because it arrived less than 2.5 s before the signal and Node's row was not there at 2 s) gets one line of its route with "outcome":"stopped" and "reason":"before_retry" (or in_handler, before_readback), carrying only the request facts its shed line carries and nothing the observation had found, and the stop ends about 7.7 s after the signal, exit 0;
    • one shadow_stopped line gives the counts (kept, stopped by where, connections still being read). No copy it answered goes without a line, and none gets two.
    • Compared with before: the old process printed every copy's line, the second-look class included, because SIGKILL came at 10 s; now a second-look copy that arrived in the last 2.5 s before the signal gets a stopped line instead (with copies arriving evenly, one in four of that class at a stop). The old process also kept accepting copies for those 10 s; now a copy nginx sends after the signal finds the listener closed, is not observed, and is counted against coverage by the comparison tool, as a down shadow is.
  • The read shadow (polis-api-shadow-reads). The same, with its own lines: a read's observation prints its line when its handler ends (its deadline is 75 ms), so it exits within a fraction of a second.
  • The whole teardown is shorter by however long its slowest container now takes less; only each polis-api container's own stop was timed.

The comparison tool (polis-shadow)

  • A stopped line is counted in a bucket of its own: never an observation, not admitted, not a would-serve, hand-back or failure, not read back, never an agreement or a disagreement, and not covered (such a copy counts against coverage, as a lost line would). In the comment report a stopped copy that has a Node access line is still counted in the paired figure (one nginx line, one Node line, one observation line), and in no judged class under it.
  • The vote report prints the stopped count, and where they were, only when there are any; over logs with no stopped line it is unchanged, byte for byte.
  • The comment report knows the word (it no longer withholds its verdict for it) and its outcome table gains a stopped row, printed as stopped 0 when there is none: its bytes change for every window, stopped lines or not. (A commit message in this branch says the reports are unchanged byte for byte over logs with no stopped line; that holds for the vote report only.)
  • The new counts are sums of an equality on outcome, a field every shadow line carries, and a conjunction in the second stage; they have none of the or over a possibly absent field that Logs Insights evaluates as empty inside sum().

A request head polis-api cannot parse (once the vote route is routed to Rust)

  • Before: dropped with no answer. nginx then answered its own 502 and the vote was never recorded, where Node records it today. Two such heads reach polis-api through nginx: a header value that is not UTF-8 (a byte above 127) and more than 128 header lines; Node accepts both.
  • After: answered 502 and closed. On POST /api/v3/votes it carries the same X-Polis-Vote-Dispatch: not-started marker as the parser's other hand-backs (chunked and oversized bodies), and on POST /api/v3/comments the comment marker, so nginx gives the whole request to Node. Nothing is dispatched. Any other path gets the 502 unmarked, which a read location's fallback sends to Node, as it did the dropped connection.

The writes the stop covers

A vote, comment, Treevite wave, Treevite acceptance or login, or topic-selection write that has not been dispatched when the stop begins is not dispatched, and nothing is written. A vote or comment gets the marked 502 hand-back that nginx's write location sends to Node. The other four answer as they do when they cannot dispatch: a wave and a topic selection 502, unmarked (no nginx location sends a wave POST, a Treevite acceptance or a topic-selection write to polis-api: their locations route only GET and HEAD without a body, or there is none); an acceptance or login 503 polis_err_treevite_unavailable. All but votes and comments are switched off in production, and votes and comments are not routed to Rust. A write dispatched before the stop is counted until it ends, and the stop waits for it, at most 5 s. What that means for a vote with the writer on is the open condition above.

What does not change

  • Every request whose head parses gets the same bytes as before, pca2 included. (One pca2 case, a math row whose document has no lastVoteTimestamp, is answered with the time of the read in that field, so its bytes differ between any two reads, on one container or two, before and after this change alike; that code is not touched here.)
  • The shadows' lines for copies that complete normally are the same, key for key and value for value, the durations aside.
  • No participant request takes a new lock, pool, query or wait. The stop's waits read counters every 10 ms and hold no database connection of their own; an observation in flight holds only what it already held. The shadow takes one more uncontended mutex before it answers a copy.
  • No vote value is written or logged by anything new; no line carries a vote, a participant, a statement, a text or a token.
  • No setting, schema, Node, nginx, Compose or deploy-hook change.

Evidence

The previous image (edge) against this one, each built from queue-rs/polis-api/Dockerfile, run as Compose runs them (read-only, cap_drop: ALL; the pca2 server at 0.5 CPU and 512 MB, the shadow at 0.25 CPU and 384 MB) against one database:

  • docker stop, PID 1, idle, three runs each: the pca2 server 10.18-10.21 s, exit 137 before; 0.11-0.15 s, exit 0 after. The shadow 10.15-10.19 s, exit 137 before; 0.13-0.15 s, exit 0 after.
  • docker stop with copies in flight in the shadow: with Node's row there at the first look, every copy's own line, 1.64-1.66 s, exit 0 (before: every line, 10.22-10.24 s, exit 137); with Node's row absent, a copy that arrived 3.5 s before the stop printed its own line at its 10 s look and the copies that arrived 0.5 s before got stopped lines, 7.69-7.73 s, exit 0 (before: every line, 10.22-10.24 s, exit 137).
  • On a loaded box (this change, merged with the current development branch): the shadow at its Compose limits and pinned to two CPUs, with eight busy processes in two other containers on the same two CPUs, 20 stops each with seven copies whose Node rows were there (1.60-1.65 s), with nine copies of which seven were waiting for their second look (7.64-7.69 s), and with thirteen copies all waiting (7.65-7.68 s): 60 of 60 exit 0, 580 copies, each with exactly one line (none lost, none twice, none cut short).
  • pca2 and the other well-formed requests, 16 of them, Date removed: 15 identical; the sixteenth is the case above whose answer carries the read time.
  • The shadow's lines for eight copies that complete normally (votes and comments; Node's row there at 2 s, absent until 10 s, no token, an unknown conversation): the same keys in the same order and the same value in every field but the four durations; the answers to the copies identical.
  • On this change merged with the current development branch: make check-polis-api, whole, on x86_64: 1,409 Rust tests passed, 0 failed, 2 ignored (both database passes, the stop tests included), and the image, routing and shadow suites passed. make check-vote-sign-lint: no new sign site. The comparison tool's tests: 69 passed, 0 failed. The other local checks this change triggers (E2E, server integration, the recordings, the queue and CLI suites, the vote-writer series): passed, except one whose image build timed out reaching an outside registry before any of its tests ran.
  • The vote-safety tests under load, the previous code against this change alternating run by run on a quiet machine: heavy (eight load processes, two test threads, PostgreSQL at one CPU) 100 of 100 whole-filter passes on each side, 0 failures; light (four load processes, four test threads) 30 of 30 on each side, 0 failures.

Tests

Written first; on the code before each change they fail.

  • queue-rs/polis-api/tests/stopping.rs runs the real binary against its own database:
    • SIGTERM while one vote is held inside its transaction and another is still arriving: the first commits and is answered, the second is handed back (marked) and not written, and the process exits zero before its bound.
    • SIGTERM while a committed vote's next-statement selection cannot finish: the caller was answered at the one-second deadline, the process waits its 5 s bound and exits zero in about 5.5 s, and the vote is written once.
    • Both roles exit zero at once when idle.
    • 30 votes and 10 reads queued in the kernel's backlog at SIGTERM: every one answered, the votes with the marked hand-back and none written.
    • The shadow, SIGTERM with three votes and two comments whose Node rows are there at the first look: all five lines printed, exit about 1.5 s after the signal. Before: no line.
    • The shadow, SIGTERM with copies waiting for the second look: the two that arrived 3.5 s before the signal are kept, the three that arrived 0.5 s before get stopped lines at the bound, exit zero inside the window. Before: no line.
  • Unit tests: the shadow's registry gives each copy exactly one line under racing threads; the drain; each route's stopped line against its line contract; the stop's forced interleavings (a dispatch paused between its count and its read of the stop, the other order, the swapped order as a mutant the harness detects, a stress race); the accept loop's sweep of connections queued at the stop; a stopping process dispatches no Treevite acceptance (before: 201, a participant written) and no topic-selection mutation (before: 200), and a dispatched one is counted until it ends.
  • http.rs: every head that fails to parse, on the vote path, gets exactly the bytes of the chunked-vote hand-back and never reaches the handler; on the comment path the comment marker; on other paths an unmarked 502 with Connection: close.
  • polis-shadow: stopped lines are counted apart and leave every criterion's mark and the verdict as they were.

Not in this change

  • The open condition above (a vote in flight at a stop with the writer on): stated, not fixed.
  • Keeping every shadow copy that waits for its second look would need a longer stop window for polis-api-shadow (a stop_grace_period of 15 s and a bound of 11 s); this change fits today's 10 s.
  • Whether a box is taken out of the load balancer before its containers stop.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

🤖 Generated with Claude Code

https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

colinmegill and others added 7 commits November 18, 2026 18:45
…ndispatched one is handed back, and the process exits inside the stop window

What changes for someone stopping the process (docker stop, docker compose
down, the web deploy's teardown):

- polis-api handled SIGINT only. As PID 1 in its container it ignored
  SIGTERM, so every stop waited the full 10 s and ended in SIGKILL (exit
  137); outside a container SIGTERM killed it at once. Either way a vote
  inside its transaction got no answer.
- Now SIGTERM and SIGINT are handled alike, installed before any startup
  work. The main role stops accepting connections, serves the connections
  the kernel had already completed, and waits until no request is being read
  or answered, no dispatched write is running and no notification is queued,
  at most 5 s. A vote, comment or wave write not yet dispatched when the stop
  begins is not dispatched: it is answered with the marked 502 hand-back
  nginx sends to Node. Blocking work still running afterwards is waited for
  at most 0.5 s, so a stop ends within about 5.5 s, inside Compose's 10 s.
  Two lines say what happened: polis_api_stopping (the signal) and
  polis_api_stopped (whether it drained, how long, what it left, and how many
  writes it handed back).
- The shadow role stops at once on SIGTERM or SIGINT (it writes nothing) and
  prints shadow_stopping.
- Nothing changes for any request while the process is not stopping: no new
  lock, pool, query or wait on a request; the wait reads counters and holds
  no database connection.

Tests, written first and failing on the previous code (SIGTERM ended the
process with no answer and no exit status), run the real binary against its
own database (tests/stopping.rs): a vote held inside its transaction at
SIGTERM commits and is answered while a vote still arriving is handed back
and not written, and the process exits zero before its bound; a committed
vote whose next-statement selection cannot finish is answered at its
deadline, and the process waits the bound and exits zero in about 5.5 s; and
both roles exit zero at once when idle.

One existing test reads main.rs as text up to the `#[tokio::main]`
attribute, which main no longer carries (it now builds the runtime itself
so that blocking work is waited for at most 0.5 s at exit). Its anchor is
now the start of `fn main`: the same span of source, the same keys.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
… 502 hand-back, never dropped, so a vote with such a head reaches Node

What a participant would observe once POST /api/v3/votes is routed to
polis-api: a vote whose request head polis-api could not parse was dropped
with no answer, so nginx answered its own 502 and the vote was never
recorded, where Node records it today. Two such heads reach polis-api
through nginx: a header value that is not UTF-8 (a byte above 127) and more
than 128 header lines; Node accepts both.

Now the main listener answers every head it cannot parse with a 502 and
closes the connection. On POST /api/v3/votes it carries the same
X-Polis-Vote-Dispatch: not-started marker as the parser's other hand-backs
(chunked and oversized bodies), and on POST /api/v3/comments the comment
marker, so nginx's vote location gives the whole request to Node, which
reads it with its own parser. Nothing is dispatched: no body is decoded and
no handler runs. Any other path gets the 502 unmarked, which a read
location's fallback sends to Node, as the dropped connection did. This
covers every way a head fails in the parser: a value that is not UTF-8,
more than 128 header lines, a control byte in a value, a header name that
is not a token, a Content-Length that is not a number or two that differ,
an unknown HTTP version, and a head longer than 1 MB. The log line for the
refusal is unchanged. The shadow listener is unchanged: it still answers
nothing and counts the refusal. A request whose head parses gets exactly
what it got before.

Tests in http.rs, written first and failing on the previous code (the reply
was empty): every such head on the vote path gets the same bytes as the
chunked-vote hand-back, and the handler never runs; on the comment path the
comment marker; on any other path an unmarked 502 with Connection: close.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…ad-heads

No conflicts: edge changed nothing under queue-rs/polis-api.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…, and prints a stopped line for each one still in flight at its bound

The shadow role exited as soon as SIGTERM arrived, so every deploy lost
the observation line of each copy still being observed: a vote or comment
copy prints its line only after its read-back of what Node stored, 2 s
after the copy arrived, or 10 s after when Node's row is not there at the
first look. Inside its container the old process ignored SIGTERM as PID 1
and the ten seconds before SIGKILL printed them; the change before this
one lost them, the rarest class most of all, and counted nothing.

Now, on SIGTERM or SIGINT, the shadow stops accepting copies and waits at
most 7.5 s (shadow_stop.rs) for the observations it has begun. Each
prints its own line as before. At the bound, each observation still in
flight gets one line of its route, every key in its order, with outcome
"stopped" and where it was in reason (in_handler, before_readback,
before_retry), carrying nothing the observation had found: no would-be
answer, admission, conversation, duration or read-back, as on a shed
line. Then one shadow_stopped line gives the counts: kept, stopped by
where, and connections still being read. Exactly one line per copy: the
observation's own or its stopped line, whichever claims it first. A copy
registered after the stop has printed its lines gets its stopped line at
once.

The window is Compose's default stop_grace_period, 10 s, which none of
the three polis-api services changes, and the web deploy's teardown
(--profile rust-api down) gives no other. 7.5 s leaves 2.5 s for the
runtime's half second, the lines and a quarter CPU's scheduling. It keeps
every copy whose Node row is there at the first look, and of those
waiting for the second look the ones that arrived at least 2.5 s before
the stop: three in four with copies arriving evenly.

The wait holds no database connection: it reads two counters every
10 ms. A read-back marks the wait for its second look through a
task-local; outside an observation it does nothing. A copy that completes
normally prints the same line as before, byte for byte.

Tests, written first against the previous binary (no line was printed
for any copy in flight at SIGTERM): with the real binary, three votes and
two comments read back at 2 s are all printed and the process exits about
1.5 s after the signal; a vote and a comment that arrived 3 s before the
signal are kept at their 10 s look, and two votes and a comment that
arrived 0.5 s before get stopped lines at the bound, exit zero inside the
window. Unit tests of the registry (one line per copy under racing
threads), the drain, and each route's stopped line against its line
contract. Also the independent review's test that every connection queued
in the kernel's backlog at SIGTERM is answered.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…, never as observed, covered or judged

The shadow now prints a line with outcome "stopped" for a copy it was
still observing when it was stopped, past its bound. The comparison tool
read an outcome it did not know as nothing in the vote report (absent
from the outcomes, yet counted as covered in the join) and as an
unrecognised word in the comment report, which withholds the verdict.

Now, for both routes: the join counts stopped copies on their own
(stopped) and no longer as covered, so such a copy counts against
coverage as a lost line would; no would-serve, hand-back, failure,
admitted, read-back, pairing or disagreement class includes it. The vote
report prints the stopped count and where they were when there are any;
the comment route knows the word and its three reasons and prints them
with the other outcomes. Reports over logs with no stopped line are
unchanged, byte for byte (the six vote reports pinned as fixtures).

Tests: the passing week with three stopped vote copies added, and the
passing comment fixture with two, keep every criterion's mark and the
verdict; the join's stopped count is the added copies and every other
class is unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…d by the stop, and the stop's tests each hold a stop of their own

Two writers, both switched off in production and routed to polis-api by
no nginx location, spawned their write as a detached task that the
drain did not count: a Treevite acceptance or login (treevite_write.rs)
and a topic selection's mutation (topic_selection_writes.rs). After
their reply deadline (1 s and 1.1 s) such a write ran on with nothing to
wait for it at a stop. Both now take the stop's count just before the
spawn and hold it in the task, as votes, comments and waves do. A
stopping process dispatches neither: the Treevite route answers 503
(polis_err_treevite_unavailable), its answer when it cannot take a
dispatch; the topic-selection route answers 502, its answer before every
mutation. Nothing is written either way.

The wave route's stop answer is a 502 without the vote writer's
hand-back marker, like its other answers before dispatch: no nginx
location sends a wave POST to polis-api (the treeviteWaves location
routes only GET and HEAD without a body), so a marker would be read by
nothing. The commit that added the stop said otherwise; the code says so
now.

The stop is now a value: the process holds one, and a test holds its
own (Stop::for_test, scoped), so a test can stop without handing back
the writes of the other tests in its process. With it, the independent
review's forced-interleaving tests run in the ordinary suite: a dispatch
paused between its count and its read of the stop (the drain waits, the
request is handed back), the other order (the drain waits for the
dispatch), the swapped order as a mutant the harness detects, and a
stress race (200 rounds of 4 threads by default; RACE_ROUNDS and
RACE_THREADS raise it). And the review's test of the accept loop's
sweep: every connection queued when the stop comes is answered, and one
after it is refused. Nothing the process does changes.

Tests, written first: against the handlers without this change, a
stopping process accepted the invite (201, a participant written) and
mutated the selection (200); now 503 and 502 and nothing written, and a
dispatched acceptance is counted past its 202 until it commits, a
dispatched mutation while it runs.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
No conflicts. The edge side changes the Delphi shadow's job graph and its
start in the deploy hook; nothing under queue-rs/polis-api, nginx or the
rust-api profile's teardown.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant