native: split the tic into two resident statements - #521
Merged
Conversation
github-actions
Bot
added
area: driver
MarcusKainth
marked this pull request as ready for review
September 7, 2026 15:49
MarcusKainth
force-pushed
the
native/two-resident-statements
branch
from
September 7, 2026 18:04
bb78e82 to
033a780
Compare
Analysis on the tic's own resident statement cost 58.4 s, and most of it was not accounted for by any routine's own arithmetic: the final projection re-lists every still-needed binding at every nesting level a statement this deep has, and that repetition is what QueryAnalysisMicroseconds was paying for, not the stages themselves. A copy of the generator with hud::tickers and the melt removed still cost close to the same; narrowing the outer projection to five columns did not either. Splitting the tic in two at the thinker/specials boundary cut analysis to 26.2 s combined, and the two analyse concurrently at session start, so a session's first tic also drops by more than half. The first statement carries the player and the thinkers and writes one row per tic into a new table, native_stage, shaped like native_state plus two scratch columns the second statement reads by their own bare name across the boundary: px_crossed_line and tx_crossed_line, neither a native_state column. The second statement reads that row back through joinGet the way the whole tic reads the tic before, and writes native_state as before. Everything else scratch-named at the boundary resolves through state or is a map constant both stages reload fresh from the same reference tables, so native_stage does not need to carry it. native_stage's own column list duplicates native_state's: a Join engine table refuses ALTER TABLE ... ADD COLUMN, so the two cannot share a declaration, and a change to one's contract is a change to both. The schema check covers the new table the same way it covers every other one. Unit tests: the two statements' own column sets meet exactly at the boundary, and no bare alias but the two staged ones crosses it. Every test that drives run_statement or demo_statement over more than one row now gets both statements back and issues them with .extend rather than .push, which is the minimum a caller needs to keep compiling against the new return shape; native/tests/support/resident.rs, in a later commit, is what actually feeds them the way a session does. NATIVE.md's resident statements section describes both statements and native_stage in the same commit as the generator that implements them.
A resident statement's wire protocol, the padding row, the chunked body,
the settings a statement needs before the server parses it, is part of
native mode's own contract, not something the driver invents on its own
behalf. It moves from driver/src/native/{stream,url,rowbinary,settings}.rs
to native/src/resident/, verbatim apart from Endpoint replacing the
driver's own ConnArgs at the open boundary, since the native crate cannot
depend on the driver's clap-derived connection arguments. The driver's
own Session converts ConnArgs to Endpoint at the one place it opens a
statement.
native/src/lib.rs's own doc comment carves out this exception: everything
else in the crate builds SQL text and executes nothing, and resident is
the one part that does, because the wire protocol it runs on is itself
part of the contract NATIVE.md states.
driver/tests/native_stream_live.rs stays in the driver crate: it is
native::stream's own live proof rather than a session-level one, and
moving it would mean rewriting its Db-based read-back onto native's own
Fixture for no behavioural reason.
The driver's own cli/native.rs and native/session.rs pick up the Endpoint
conversion at their own open call sites. session.rs's own two-statement
handoff is a separate commit.
Session::open sends both simulation statements at once, alongside the renderer, so a session's first tic pays for the larger of their analyses rather than the sum. wait_sim feeds the second statement the moment native_stage holds the first's own row for a tic, then goes on polling native_state the way it always did, so a caller that only calls feed_sim and wait_sim sees no difference from one statement. recover ends and reopens both, and reports the error either one gave. Session::open and Session::recover both truncate native_stage before either simulation statement opens. The table never carries anything past the tic in flight when its own writer last ran, so a session pays for one recomputed tic rather than risk a row a prior run left behind answering the first statement's own presence check for a tic the reopened statement has not written itself. native play resumes straight from whatever native_state and native_stage hold across process runs and never truncates on its own, so it depended on this. demo's own restart_sim and diff's own restart already truncate native_state for a full reset and now truncate native_stage alongside it, for the same reason at the point a run chooses to start over rather than resume. Ending a session's two simulation statements is sequential, not concurrent: the second statement's own body includes rows that read the first's, so ending them at once risked the second waiting on a commit the first's own close had not yet forced. SessionError::SimClose names which of the two failed to close, since a caller debugging a hang needs to know which; SessionError::RenderClose carries the same for the renderer. CLOSE_TIMEOUT widens from 10 s to 60 s: it bounds detecting a statement that has died after its own body closed, not the cost of a normal close, and a close happens once per run. When a close does time out, the error reads system.processes for the statement's own query id, and system.query_log for it once the process is gone, and carries what it finds rather than leaving the next person to reproduce it by hand.
A batch that runs every tic through the first statement and then every tic through the second is wrong past the first tic: the first statement's own previous(db) reads the tic before out of native_state, which the second statement writes, so a tic beyond the first reads a row the second statement has not written yet. native/tests/support/resident.rs opens both statements once, the way a session does, feeds one tic to the first, waits for its own row in native_stage, feeds the second from it, and waits for its own row in native_state before feeding the next tic. Confirmed live against sim_door_live's own door crossing before this existed: a door spawned correctly on the crossing tic and then lost every thinker the very next tic, because the first statement's own batch, still computing later tics, read native_state before the second statement's batch had ever written any of it. Every sim_*_live.rs suite that ran more than one tic through run_statement or demo_statement switches to it. A suite that seeds one row and runs exactly one tic from it keeps the batched call, since a batch of one row has no later tic to read a stale block for. sim_thrust_live's own "crossed" arm surfaced a real, if unrelated, gap in its own seeding once the tics it copies from ran correctly for the first time: it gives every line special 88 so the arm does not depend on the map putting one where the thrust happens to go, and every untouched line keeps tag 0. P_FindSectorFromLineTag compares tags as plain numbers, so tag 0 matches every untagged sector in the map, not "no sector", and the crossing spawned a plat on every one of them at once rather than the one sector the arm means to check. The arm now gives every line and one sector the same otherwise-unused tag, isolating the spawn to that sector, and reads its own thinker directly to say so. The same fix corrected the arm's own stale assertion: 88 is in specials::PLAT_TRIGGER_SPECIALS, which unhandled_crossable filters out of the unresolved set, so a clean crossing leaves nothing unresolved, not TX_CROSSED as the arm's own comment and assertion had said.
FIRST_TIC_TIMEOUT and TIC_TIMEOUT moved into native::resident, beside CLOSE_TIMEOUT: all three bound how long a caller waits on a resident statement, which is a fact about the protocol rather than about the driver's own session or a test harness. session.rs no longer defines them; its callers (demo, diff, play, and their own live test) read them from native::resident, and wait_sim's own doc comment points there too. The test harness keeps its own TIC_TIMEOUT at 30 s rather than the driver's 5 s, since a test has no 35 Hz budget to pace against and a wide bound costs nothing on a shared machine; it reuses FIRST_TIC_TIMEOUT unchanged, since that one is sized for a slow CI runner either way. FIRST_TIC_TIMEOUT carries main's own current value, 600 s, not the 300 s this branch had been carrying: #506 raised it after three lanes' own containers running together measured about 59 s for the tic's own statement on main, well past 300 s on a CI runner four times slower. The bound still catches a dead statement either way; the cost itself is tracked separately from what this constant is for. tick::resident_statements(db) pairs the tic's own two statements in one call, and resident_statement_stage1/stage2 drop to pub(crate): the second statement's own input depends on rows only the first writes, so nothing outside the crate can open one without the other, which is the property NATIVE.md states. Every caller that built the pair by hand (the three CLI subcommands and their own live tests, the harness) now calls it instead.
sim_player_damage_live.rs and sim_refire_live.rs, sim_gunshot_kill_drop_ live.rs, sim_kills_and_drops_live.rs, sim_missile_kill_drop_live.rs, and sim_missile_same_target_live.rs and sim_missile_wall_live.rs all landed on main after this branch's split and never went through the mechanical push-to-extend fix every other caller of run_statement and demo_statement needed for the new two-statement return shape, since the two-statement generator did not exist yet where they were written. sim_thrust_live.rs's later "crowd" arm carried the same gap in its single-tic run_statement call. sim_refire_live.rs, sim_gunshot_kill_drop_live.rs and sim_missile_wall_ live.rs seed every arm straight from tic 0 and run exactly one tic from it, so the mechanical fix is all either needed. sim_player_damage_live.rs's, sim_kills_and_drops_live.rs's, sim_missile_kill_drop_live.rs's and sim_missile_same_target_live.rs's BEFORE-walk is the same arms-style multi-tic pattern the previous commit already moved seven other suites through, so it gets the same treatment here.
MarcusKainth
force-pushed
the
native/two-resident-statements
branch
from
September 7, 2026 19:03
033a780 to
bb8bfbf
Compare
The crossing arm's own tag isolation (a shared tag on every line and one sector, so a plat special's tag-0 search would land on one sector rather than every untagged one) and #499's own fix for the identical problem (switching the arm to a special native does not dispatch at all, so no search runs) both solved the same tag-0 storm this arm hit once type 22 made 88 a dispatched special, on separate branches, at separate times. Rebasing onto main picked up #499's own rename to MONSTER_UNHANDLED at the one line both patches touched, and kept the tag isolation and the plat- spawn assertions this branch had added around it: git's own merge saw no overlapping line to conflict on, so the result carried a special that never dispatches anything, an ALTER TABLE pair tagging sectors nothing ever reads, and an assertion still expecting the dispatch to run and leave nothing unresolved. CI caught it: `unresolved` came back as `TX_CROSSED`, since the crossing genuinely is not run. MONSTER_UNHANDLED is undispatched by design, so nothing needs isolating from it. The tag mutations, `CROSSING_TAG`, `CROSSING_SECTOR` and the `plat_at_crossing_sector` column drop; the arm's own final assertion goes back to what #499 already established, that the crossing leaves the tic unresolved with `TX_CROSSED` and nothing else.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased twice since review. First onto
mainafter #511 and #509 bothlanded (head 033a780):
git cherry-pickof the first four commits appliedclean, the fifth conflicted because main independently gained a
FIRST_TIC_TIMEOUT/TIC_TIMEOUTshare (a different PR did roughly the sameconsolidation before this branch started) and a later PR raised
FIRST_TIC_TIMEOUTto 600 s under real multi-lane CI load; resolved bykeeping main's own 600 s rather than the 300 s this branch had been
carrying, with the reasoning in that commit's own message rather than the
code. A sixth commit migrated five files #505 and #509 had added after this
branch's own split, which never went through the mechanical
.push-to-.extendfix every other caller needed.Then onto
mainagain after #513 landed (head bb8bfbf): the same fivecommits applied clean this time, since main already carried the 600 s
FIRST_TIC_TIMEOUTfrom the first rebase's own resolution. The sixthcommit picked up two more files #513 added
(
sim_missile_same_target_live.rs,sim_missile_wall_live.rs) needing thesame treatment, folded into that same commit along with a retitle (the
original named the two PRs rather than saying what the commit does) and
trimming a few "own"s the body didn't need.
What this changes, and why
The tic's own resident statement analysed at 58.4 s on a quiet run and past a
minute under load, and most of that cost was not any routine's own
arithmetic: a statement nested this deep re-lists every still-needed binding
at every level, and that repetition, not the stages themselves, is what
QueryAnalysisMicrosecondswas paying for. Removinghud::tickersand themelt, or narrowing the outer projection to five columns, left the cost
essentially where it was; only cutting the tic in two, at the boundary
between the player/thinkers and the specials/
G_Ticker, moved it.The simulation is now two resident statements chained through a new table,
native_stage: the first writes one row per tic there, through the playerand the thinkers; the second reads that row back and writes
native_state,through the specials and
G_Ticker. Both statements are sent at session openso their analyses overlap rather than add up, and the driver feeds the
second the moment the first's own row is visible, so a caller that only
calls
feed_sim/wait_simsees no difference from one statement. Theprotocol both statements run on (the chunked body, the padding row, the
settings) moves into
native's ownresidentmodule, since the wireprotocol is part of native mode's own contract and not something the driver
should carry alone. Every multi-tic test that drove the simulation through a
batched, non-interleaved call is migrated to a harness that feeds it the way
a session does, because a batch that runs every tic through the first
statement and then every tic through the second reads
native_statebeforethe second statement has written any of it past the first tic — confirmed
live against
sim_door_live's own door crossing, which lost its thinker thetic after it spawned one under the old batched call.
Evidence
Analysis, each statement and their sum, against a fresh measurement of
main's own single statement, back to back on the same server so the
comparison is fair to whatever the machine was doing at the time (this
machine runs several other sessions; absolute numbers move with load, the
ratio does not):
First tic wall time, same back-to-back run,
native demo --from sim --stop-at-frame 5: main 84.8 s, this branch 35.0 s. Both statements (andthe renderer) open at once, so the branch's own first tic pays for the
larger analysis rather than the sum; the ratio here (2.4x) tracks the
analysis ratio above rather than adding the renderer's own ~1.1 s on top.
State-row hash against main, same demo, run to the first shared refusal
(both stop at tic 177,
PSP_STUCK, since neither branch's own thinker orderchanged):
sum(cityHash64(*))over every column of every rownative_stateholds for tic 0 through 176.
Session timing, this branch, live:
wait_sim's own p50/p99 covers the full two-statement hop (feed the first,wait for
native_stage, feed the second, wait fornative_state), againstTIC_TIMEOUT's own 5 s budget.native demo --from sim, this branch, a real paced run againstdemo3:tics/s tracks what the level is doing rather than holding at 35, the same as
main; both branches reach the same first refusal (tic 177,
PSP_STUCK) atthe same tics/s profile.
Four close-timeout reproduction attempts, before the real cause was found:
sequential background load, eight concurrent
native diffprocessessharing the server (analysis times up to ~51,000,000 µs), the debug binary
in place of release (matching what
cargo test's own subprocess runs), anda quiet baseline. All four completed cleanly:
system.query_logshowedtype: QueryFinish, an emptyexception, andquery_duration_msmatchingwall clock exactly for the statement each attempt watched. None reproduced
the close hang.
The real cause was not a hang.
native_demo_live's own failing test runsclickdoom native demotwice against one database: once to frame 41 for ahash check, then again with
--stop-at-frame 500expecting the refusal atexit 3. The second run's own restart truncated
native_stateand rewrotethe level's first row, but never touched
native_stage, so the table stillheld the first run's own staged rows.
Session::wait_sim's presence checkreads
native_stagedirectly, with no notion of which run wrote a row, sofor every tic the second run asked about it found the first run's leftover
row already there and fed the second statement immediately, without ever
waiting on the second run's own first statement to write it. Feeding raced
ahead of real computation for as long as the leftover rows lasted; the
moment they ran out,
wait_simhad to wait for real production for thefirst time, the first statement had whatever backlog the race left it with,
and the run missed
TIC_TIMEOUT— the same symptom on a different callpath from the close-after-refusal timeout the four attempts above could not
catch, since none of them reused a database across two runs of the
simulation.
restart_sim(demo) andrestart(diff) now truncatenative_stagealongsidenative_state;Session::openandSession::recoveralso truncate it unconditionally, since the table nevercarries anything past the tic in flight and native play resumes across
process runs without ever calling either restart path.
CLOSE_TIMEOUTwidens from 10 s to 60 s regardless: it bounds detecting astatement that has died after its own body closed, not the cost of a normal
close, and a close happens once per run.
SessionError::SimCloseandSessionError::RenderClosename which statement failed to close and, on atimeout specifically, read
system.processesandsystem.query_logforthat statement's own query id so the next occurrence carries its own
diagnosis.
One more finding, orthogonal to the split itself:
sim_thrust_live's own"crossed" arm gives every line special 88 (a WR plat trigger) so the arm
does not have to know which line the seeded thrust crosses, and left every
other line at its own tag, 0.
P_FindSectorFromLineTagcompares tags asplain numbers, so tag 0 matches every untagged sector in the map, not "no
sector": once the tic before the arm was computed correctly for the first
time (by the interleaved harness, rather than the batch bug's own defaults),
some other sector genuinely already running a thinker fell inside the tag-0
blast radius, and the arm's own crossing spawned a plat on it too, which is
exactly what
PLANE_CLIP_STUCKexists to flag. The arm now gives every lineand one chosen sector the same otherwise-unused tag, isolating the spawn,
and reads that sector's own thinker directly rather than only inferring it
from the absence of a stuck bit. The same investigation found the arm's own
assertion was already stale before this: special 88 is in
specials::PLAT_TRIGGER_SPECIALS, whichunhandled_crossablefilters outof the unresolved set, so a clean crossing of it leaves nothing unresolved,
not
TX_CROSSEDas the arm's own comment had said.Full live pass before the rebase, one lock hold,
--test-threads=1withineach nextest group:
After the first rebase (onto main with #511/#509), against a throwaway
database on the shared server:
After the second rebase (onto main with #513), the two files it added:
Unit suites, clippy and fmt, on the final rebased head (bb8bfbf):
Invariants
None of
PURITY.md's numbered rules changes shape.native_stageis aplain
Jointable likenative_state, populated the same way; the two newfiles in
native/src/resident/carry the sameInstant::now()timing thedriver's own resident code already carried, each annotated
purity-okforthe same reason (pacing and timeouts in the caller, never a value a
statement reads).
Spec impact
SPEC.mdis touched.NATIVE.md's own residentstatements section changes in the same commit as the generator that
implements it.
Checks
make gates— not run directly; the suites above cover itmake native-smokeAnything else
Six commits, each verified to build on its own in an isolated worktree
before the next was written on top:
native: split the tic into two resident statements— the generator,native_stage's schema,NATIVE.md, the golden plan listing (shifted bythe new table), and the mechanical
.push→.extendevery caller ofrun_statement/demo_statementneeds for the new two-statement returnshape.
native: own the resident protocol as native mode's own contract— themove from
driver/src/native/{stream,url,rowbinary,settings}.rstonative/src/resident/.driver: drive the simulation as two resident statements—Session'sown two-statement handoff, the
native_stagetruncate-on-open/recover,the widened
CLOSE_TIMEOUTand the self-diagnosing close error.native: drive multi-tic tests through the two resident statements—the harness and every suite it replaces a batched call in, plus the
sim_thrust_liveseeding fix the harness surfaced.native: state each resident timeout and the statement pairing once—FIRST_TIC_TIMEOUT,TIC_TIMEOUTandCLOSE_TIMEOUTall live innative::resident;tick::resident_statementspairs the two statementsin one call, and the individual functions drop to
pub(crate).native: drive the remaining live suites through the harness— thesame mechanical and harness fix applied to seven files that landed on
main after this branch's split existed.
Written mostly by Claude Sonnet 5.