Repository navigation
polis-api: stops cleanly on SIGTERM, keeps or counts every shadow observation in flight, and answers a request head it cannot parse - #3455
Open
colinmegill wants to merge 7 commits into
Conversation
…ndispatched one is handed back, and the process exits inside the stop window What changes for someone stopping the process (docker stop, docker compose down, the web deploy's teardown): - polis-api handled SIGINT only. As PID 1 in its container it ignored SIGTERM, so every stop waited the full 10 s and ended in SIGKILL (exit 137); outside a container SIGTERM killed it at once. Either way a vote inside its transaction got no answer. - Now SIGTERM and SIGINT are handled alike, installed before any startup work. The main role stops accepting connections, serves the connections the kernel had already completed, and waits until no request is being read or answered, no dispatched write is running and no notification is queued, at most 5 s. A vote, comment or wave write not yet dispatched when the stop begins is not dispatched: it is answered with the marked 502 hand-back nginx sends to Node. Blocking work still running afterwards is waited for at most 0.5 s, so a stop ends within about 5.5 s, inside Compose's 10 s. Two lines say what happened: polis_api_stopping (the signal) and polis_api_stopped (whether it drained, how long, what it left, and how many writes it handed back). - The shadow role stops at once on SIGTERM or SIGINT (it writes nothing) and prints shadow_stopping. - Nothing changes for any request while the process is not stopping: no new lock, pool, query or wait on a request; the wait reads counters and holds no database connection. Tests, written first and failing on the previous code (SIGTERM ended the process with no answer and no exit status), run the real binary against its own database (tests/stopping.rs): a vote held inside its transaction at SIGTERM commits and is answered while a vote still arriving is handed back and not written, and the process exits zero before its bound; a committed vote whose next-statement selection cannot finish is answered at its deadline, and the process waits the bound and exits zero in about 5.5 s; and both roles exit zero at once when idle. One existing test reads main.rs as text up to the `#[tokio::main]` attribute, which main no longer carries (it now builds the runtime itself so that blocking work is waited for at most 0.5 s at exit). Its anchor is now the start of `fn main`: the same span of source, the same keys. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
… 502 hand-back, never dropped, so a vote with such a head reaches Node What a participant would observe once POST /api/v3/votes is routed to polis-api: a vote whose request head polis-api could not parse was dropped with no answer, so nginx answered its own 502 and the vote was never recorded, where Node records it today. Two such heads reach polis-api through nginx: a header value that is not UTF-8 (a byte above 127) and more than 128 header lines; Node accepts both. Now the main listener answers every head it cannot parse with a 502 and closes the connection. On POST /api/v3/votes it carries the same X-Polis-Vote-Dispatch: not-started marker as the parser's other hand-backs (chunked and oversized bodies), and on POST /api/v3/comments the comment marker, so nginx's vote location gives the whole request to Node, which reads it with its own parser. Nothing is dispatched: no body is decoded and no handler runs. Any other path gets the 502 unmarked, which a read location's fallback sends to Node, as the dropped connection did. This covers every way a head fails in the parser: a value that is not UTF-8, more than 128 header lines, a control byte in a value, a header name that is not a token, a Content-Length that is not a number or two that differ, an unknown HTTP version, and a head longer than 1 MB. The log line for the refusal is unchanged. The shadow listener is unchanged: it still answers nothing and counts the refusal. A request whose head parses gets exactly what it got before. Tests in http.rs, written first and failing on the previous code (the reply was empty): every such head on the vote path gets the same bytes as the chunked-vote hand-back, and the handler never runs; on the comment path the comment marker; on any other path an unmarked 502 with Connection: close. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…ad-heads No conflicts: edge changed nothing under queue-rs/polis-api. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…, and prints a stopped line for each one still in flight at its bound The shadow role exited as soon as SIGTERM arrived, so every deploy lost the observation line of each copy still being observed: a vote or comment copy prints its line only after its read-back of what Node stored, 2 s after the copy arrived, or 10 s after when Node's row is not there at the first look. Inside its container the old process ignored SIGTERM as PID 1 and the ten seconds before SIGKILL printed them; the change before this one lost them, the rarest class most of all, and counted nothing. Now, on SIGTERM or SIGINT, the shadow stops accepting copies and waits at most 7.5 s (shadow_stop.rs) for the observations it has begun. Each prints its own line as before. At the bound, each observation still in flight gets one line of its route, every key in its order, with outcome "stopped" and where it was in reason (in_handler, before_readback, before_retry), carrying nothing the observation had found: no would-be answer, admission, conversation, duration or read-back, as on a shed line. Then one shadow_stopped line gives the counts: kept, stopped by where, and connections still being read. Exactly one line per copy: the observation's own or its stopped line, whichever claims it first. A copy registered after the stop has printed its lines gets its stopped line at once. The window is Compose's default stop_grace_period, 10 s, which none of the three polis-api services changes, and the web deploy's teardown (--profile rust-api down) gives no other. 7.5 s leaves 2.5 s for the runtime's half second, the lines and a quarter CPU's scheduling. It keeps every copy whose Node row is there at the first look, and of those waiting for the second look the ones that arrived at least 2.5 s before the stop: three in four with copies arriving evenly. The wait holds no database connection: it reads two counters every 10 ms. A read-back marks the wait for its second look through a task-local; outside an observation it does nothing. A copy that completes normally prints the same line as before, byte for byte. Tests, written first against the previous binary (no line was printed for any copy in flight at SIGTERM): with the real binary, three votes and two comments read back at 2 s are all printed and the process exits about 1.5 s after the signal; a vote and a comment that arrived 3 s before the signal are kept at their 10 s look, and two votes and a comment that arrived 0.5 s before get stopped lines at the bound, exit zero inside the window. Unit tests of the registry (one line per copy under racing threads), the drain, and each route's stopped line against its line contract. Also the independent review's test that every connection queued in the kernel's backlog at SIGTERM is answered. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…, never as observed, covered or judged The shadow now prints a line with outcome "stopped" for a copy it was still observing when it was stopped, past its bound. The comparison tool read an outcome it did not know as nothing in the vote report (absent from the outcomes, yet counted as covered in the join) and as an unrecognised word in the comment report, which withholds the verdict. Now, for both routes: the join counts stopped copies on their own (stopped) and no longer as covered, so such a copy counts against coverage as a lost line would; no would-serve, hand-back, failure, admitted, read-back, pairing or disagreement class includes it. The vote report prints the stopped count and where they were when there are any; the comment route knows the word and its three reasons and prints them with the other outcomes. Reports over logs with no stopped line are unchanged, byte for byte (the six vote reports pinned as fixtures). Tests: the passing week with three stopped vote copies added, and the passing comment fixture with two, keep every criterion's mark and the verdict; the join's stopped count is the added copies and every other class is unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…d by the stop, and the stop's tests each hold a stop of their own Two writers, both switched off in production and routed to polis-api by no nginx location, spawned their write as a detached task that the drain did not count: a Treevite acceptance or login (treevite_write.rs) and a topic selection's mutation (topic_selection_writes.rs). After their reply deadline (1 s and 1.1 s) such a write ran on with nothing to wait for it at a stop. Both now take the stop's count just before the spawn and hold it in the task, as votes, comments and waves do. A stopping process dispatches neither: the Treevite route answers 503 (polis_err_treevite_unavailable), its answer when it cannot take a dispatch; the topic-selection route answers 502, its answer before every mutation. Nothing is written either way. The wave route's stop answer is a 502 without the vote writer's hand-back marker, like its other answers before dispatch: no nginx location sends a wave POST to polis-api (the treeviteWaves location routes only GET and HEAD without a body), so a marker would be read by nothing. The commit that added the stop said otherwise; the code says so now. The stop is now a value: the process holds one, and a test holds its own (Stop::for_test, scoped), so a test can stop without handing back the writes of the other tests in its process. With it, the independent review's forced-interleaving tests run in the ordinary suite: a dispatch paused between its count and its read of the stop (the drain waits, the request is handed back), the other order (the drain waits for the dispatch), the swapped order as a mutant the harness detects, and a stress race (200 rounds of 4 threads by default; RACE_ROUNDS and RACE_THREADS raise it). And the review's test of the accept loop's sweep: every connection queued when the stop comes is answered, and one after it is refused. Nothing the process does changes. Tests, written first: against the handlers without this change, a stopping process accepted the invite (201, a participant written) and mutated the selection (200); now 503 and 502 and nothing written, and a dispatched acceptance is counted past its 202 until it commits, a dispatched mutation while it runs. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
No conflicts. The edge side changes the Delphi shadow's job graph and its start in the deploy hook; nothing under queue-rs/polis-api, nginx or the rust-api profile's teardown. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
polis-api: stops cleanly on SIGTERM, keeps or counts every shadow observation in flight, and answers a request head it cannot parse
Changes to the Rust API process (
queue-rs/polis-api) that the vote writer (POST /api/v3/votes) needs before it can ever serve participants, and to the comparison tool that reads the shadow's lines (queue-rs/src/bin/polis-shadow). Today the vote route is not routed to Rust and the live vote writer is off; production runs this binary as the pca2 server (polis-api), as the vote and comment shadow (polis-api-shadow) and as the read shadow (polis-api-shadow-reads). No request a participant sends is answered differently.Before the live vote writer is ever switched on: an open condition
This change does not make the live vote writer safe to switch on, and does not claim to. With the writer on (
POLIS_API_POST_VOTES=1andpostVotesrouted), a stop ofpolis-apiwhile the box's nginx still sends it votes has two outcomes that must be resolved, or explicitly accepted, before any live flip:{"acceptance":"unknown","retry":"same_vote"}. Both participation clients retry a 202 once, 300 ms later, with the same body. During the stop that retry finds polis-api's listener closed (nginx sees zero bytes sent) or, if the kernel had already queued its connection, gets the marked hand-back; either way nginx gives it to Node, and Node appends a history row on every vote request. If the original also commits (during the drain, or by a COMMIT processed just before the exit), the vote has twovoteshistory rows of the same choice and onevotes_latest_uniquerow. This is the case the vote contract already names as an activation condition (a retry routed to Node appends again); a stop is one more way into it. Before this change the process ignored SIGTERM and kept accepting, so such a retry reached the writer, which does not append a same-choice replay (or, while the original still held its row, answered the retry 500 after its 100 ms lock wait).When the route into Node opens: only when this box's nginx still accepts requests while polis-api's listener is closed, for example a stop or recreate of the
polis-apiservice alone. In the web deploy's own teardown nginx-proxy is stopped in the same--profile rust-api down, and with the deploy's load-balancer traffic control a box receives no new requests while its containers stop.What would close it, each an addition to this change rather than a reversal of it: a stop window long enough for a dispatched vote's worst case (writer acquisition 5 s, connection setup 5 s, then its statements: a
stop_grace_periodonpolis-apiand a drain bound above that, about 13 to 15 s), and keeping a 202's retry off Node's append (a same-choice retry that Node does not append, or a retry routed back to the writer). None of this can happen while the writer is off.What anyone would observe
Every deploy's stop of the three Rust containers
The web deploy's teardown (
scripts/after_install.sh,--profile rust-api down, with no-t),docker stopanddocker compose downsend SIGTERM and, 10 s later, SIGKILL: Compose's defaultstop_grace_period, which none of the three services sets. Before this change each of them, as PID 1 in its container, ignored SIGTERM, ran the whole 10 s and was killed (exit 137).polis-api, main role). It now stops accepting connections at SIGTERM, serves the connections the kernel had already completed, finishes the requests it is answering, and exits 0: in about 0.15 s when idle, at most 5.5 s. A pca2 request after that finds the listener closed and nginx's existing fallback answers it from Node (the same bytes), where before it reached a process that was about to be killed. New lines:polis_api_stopping(which signal) andpolis_api_stopped(whether it drained, how long it took, what it left, how many writes it handed back).polis-api-shadow). It now stops accepting copies at SIGTERM and waits at most 7.5 s for the observations it has begun to print their lines. A copy's line follows its read-back of what Node stored, 2 s after the copy arrived, or 10 s after when Node's row is not there at the first look. So:"outcome":"stopped"and"reason":"before_retry"(orin_handler,before_readback), carrying only the request facts its shed line carries and nothing the observation had found, and the stop ends about 7.7 s after the signal, exit 0;shadow_stoppedline gives the counts (kept,stoppedby where, connections still being read). No copy it answered goes without a line, and none gets two.polis-api-shadow-reads). The same, with its own lines: a read's observation prints its line when its handler ends (its deadline is 75 ms), so it exits within a fraction of a second.The comparison tool (
polis-shadow)stoppedline is counted in a bucket of its own: never an observation, not admitted, not a would-serve, hand-back or failure, not read back, never an agreement or a disagreement, and not covered (such a copy counts against coverage, as a lost line would). In the comment report a stopped copy that has a Node access line is still counted in thepairedfigure (one nginx line, one Node line, one observation line), and in no judged class under it.stoppedrow, printed asstopped 0when there is none: its bytes change for every window, stopped lines or not. (A commit message in this branch says the reports are unchanged byte for byte over logs with no stopped line; that holds for the vote report only.)outcome, a field every shadow line carries, and a conjunction in the second stage; they have none of theorover a possibly absent field that Logs Insights evaluates as empty insidesum().A request head polis-api cannot parse (once the vote route is routed to Rust)
POST /api/v3/votesit carries the sameX-Polis-Vote-Dispatch: not-startedmarker as the parser's other hand-backs (chunked and oversized bodies), and onPOST /api/v3/commentsthe comment marker, so nginx gives the whole request to Node. Nothing is dispatched. Any other path gets the 502 unmarked, which a read location's fallback sends to Node, as it did the dropped connection.The writes the stop covers
A vote, comment, Treevite wave, Treevite acceptance or login, or topic-selection write that has not been dispatched when the stop begins is not dispatched, and nothing is written. A vote or comment gets the marked 502 hand-back that nginx's write location sends to Node. The other four answer as they do when they cannot dispatch: a wave and a topic selection 502, unmarked (no nginx location sends a wave POST, a Treevite acceptance or a topic-selection write to polis-api: their locations route only GET and HEAD without a body, or there is none); an acceptance or login 503
polis_err_treevite_unavailable. All but votes and comments are switched off in production, and votes and comments are not routed to Rust. A write dispatched before the stop is counted until it ends, and the stop waits for it, at most 5 s. What that means for a vote with the writer on is the open condition above.What does not change
lastVoteTimestamp, is answered with the time of the read in that field, so its bytes differ between any two reads, on one container or two, before and after this change alike; that code is not touched here.)Evidence
The previous image (edge) against this one, each built from
queue-rs/polis-api/Dockerfile, run as Compose runs them (read-only,cap_drop: ALL; the pca2 server at 0.5 CPU and 512 MB, the shadow at 0.25 CPU and 384 MB) against one database:docker stop, PID 1, idle, three runs each: the pca2 server 10.18-10.21 s, exit 137 before; 0.11-0.15 s, exit 0 after. The shadow 10.15-10.19 s, exit 137 before; 0.13-0.15 s, exit 0 after.docker stopwith copies in flight in the shadow: with Node's row there at the first look, every copy's own line, 1.64-1.66 s, exit 0 (before: every line, 10.22-10.24 s, exit 137); with Node's row absent, a copy that arrived 3.5 s before the stop printed its own line at its 10 s look and the copies that arrived 0.5 s before got stopped lines, 7.69-7.73 s, exit 0 (before: every line, 10.22-10.24 s, exit 137).Dateremoved: 15 identical; the sixteenth is the case above whose answer carries the read time.make check-polis-api, whole, on x86_64: 1,409 Rust tests passed, 0 failed, 2 ignored (both database passes, the stop tests included), and the image, routing and shadow suites passed.make check-vote-sign-lint: no new sign site. The comparison tool's tests: 69 passed, 0 failed. The other local checks this change triggers (E2E, server integration, the recordings, the queue and CLI suites, the vote-writer series): passed, except one whose image build timed out reaching an outside registry before any of its tests ran.Tests
Written first; on the code before each change they fail.
queue-rs/polis-api/tests/stopping.rsruns the real binary against its own database:http.rs: every head that fails to parse, on the vote path, gets exactly the bytes of the chunked-vote hand-back and never reaches the handler; on the comment path the comment marker; on other paths an unmarked 502 withConnection: close.polis-shadow: stopped lines are counted apart and leave every criterion's mark and the verdict as they were.Not in this change
polis-api-shadow(astop_grace_periodof 15 s and a bound of 11 s); this change fits today's 10 s.🤖 Generated with Claude Code
https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
🤖 Generated with Claude Code
https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs