Repository navigation
Promote edge to stable: Polis 2.45.3 - #3454
Draft
github-actions[bot] wants to merge 2 commits into
Draft
github-actions[bot] wants to merge 2 commits into
github-actions[bot] wants to merge 2 commits into
Conversation
…go and admitted again, publication is retried across polls, and a copied host identity no longer confirms a renewing attempt (#3452) * polis-jobs: a host-bound journal entry from another boot is confirmed only after its attempt's lease has expired The graph executor binds its retained journal to a host identity file. The binding is that file's contents, so a restored snapshot of the volume, or a copied or template-baked identity file, passes it on a second machine; the boot id then differs and the entry read as provably gone, so the copy recorded a live attempt's exit as confirmed while the original's child ran. Now, for a host-bound journal only, an entry written under another boot is confirmed once the database shows its attempt's lease over: while the attempt still holds the job, when its locked_until has passed by database time; once the job is no longer held by it, one whole lease after this daemon first read it so. A live original keeps renewing its lease, so it is never confirmed; an honest reboot waits at most one lease. Waiting entries are read again at the reaper cadence, before the reaper. The read is the existing pq_job_status together with the database clock, in one read-only statement; no SQL function or table changes. Daemons without a host binding (the full-pipeline shadow, the worker boxes) recover exactly as before. New graph_recovery reasons: another_boot_lease_live, another_boot_lease_expired. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * deploy: invalid graph executor settings refuse the graph executor only; the full-pipeline shadow still starts With DELPHI_SHADOW_GRAPH_ENABLED=1, the Delphi deploy checks the graph executor's composed settings (scripts/check-shadow-graph.py). A refusal used to stop both shadows, so a typo in an optional executor's setting also stopped the full-pipeline shadow. Now it refuses the executor alone: the full-pipeline shadow, whose own settings were read back first, starts as it would with the executor switched off. The message names what is not started. Nothing changes unless DELPHI_SHADOW_GRAPH_ENABLED=1 and the shadow are both selected. Two deploy-hook tests: valid settings start the executor before the shadow; invalid settings start the shadow without the executor. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * delphi shadow: an unclaimed graph is let go after ten minutes, publication is retried on later polls, and a held graph admission says why Three changes to the scheduler's graph path, which runs only for the conversations DELPHI_SHADOW_GRAPH_CONVERSATIONS lists (empty by default): - An admitted graph none of whose nodes is ever claimed (no graph executor running, unstartable, or refused at deploy) waited the whole job deadline (default four hours) per listed conversation, holding the full-pipeline shadow. After ten minutes with every node still queued and no attempt at all, its nodes are now cancelled with the existing fenced call, its scope is released by the existing reconciliation, it is counted failed, and the full pipeline goes on (graph_unclaimed). A graph any of whose nodes was claimed is waited for as before. - Publication of a finished graph was tried three times back to back, then counted failed. Now it is tried once per poll, up to three times, the count carried in the flight (in the state file, so a restart keeps it); a transient failure keeps the flight (graph_publish_retry). A given-up publication is counted failed once: recovery no longer takes the graph up again (which reset the count and counted the failure again every poll), and its snapshot is not remembered as seen, so a later build or change reconsiders the conversation and the same snapshot's admission answers with that graph, whose publication alone runs again. - While a deferred graph holds graph admission, the scheduler says so once, with the deferred graphs' ids (graph_admission_held, reason graph_deferred). With the list empty none of this runs and the state file is unchanged (the new fields are omitted when empty). The existing test of a transient publication failure now expects the retry on the next poll, as designed; its assertions on one success, two calls and one generation are kept. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * docs: the graph executor's recovery from another boot, its deploy refusal, an unclaimed graph and publication retries Describes, in the same-host graph executor section of the configuration docs, what the three preceding changes do and the lines they write, and that an older scheduler ignores the publication count and the given-up publications on a rollback. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * deploy: a refused graph executor starts the scheduler with its graph list blank, so no graph is admitted that nothing could claim With the graph executor switched on and its composed settings refused by the deploy check, the hook started the full-pipeline shadow with the scheduler's graph list as the environment document gave it. The scheduler then admitted each listed conversation's graph, which nothing could claim. Now the shadow's start is given one more Compose file, written by the hook (docker-compose.shadow-graph-off.yml), that composes the scheduler's DELPHI_SHADOW_GRAPH_CONVERSATIONS blank: the scheduler's existing off path, so it admits no graph for that deploy. The hook reads the composed list back first; should it not be blank, the shadow is not started at all, as when its own settings are refused. One line beginning "Error:" says which, naming the refused settings (the validator prints names only) and the list by key, never a value. The full-pipeline shadow starts as before. Nothing changes unless DELPHI_SHADOW_GRAPH_ENABLED=1 and the shadow are both selected and the executor is refused: the shadow's start otherwise runs the same command with the same files. Tests: the refused case starts the shadow with the file, reads the list back, writes one line and prints no value; a list that cannot be composed blank starts no shadow; the valid case writes no file; the read-back's jq program with the real jq. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * delphi shadow: a graph let go as unclaimed holds graph admission until the next build or an hour, doubling to a day, then its conversation is admitted again as a new graph A graph let go because nothing claimed it kept its snapshot remembered as seen, and the same snapshot's admission key answered with the cancelled graph, so an unchanged listed conversation was never admitted again, with no line saying so. Now, when a graph is let go as unclaimed (decided before its cancel, or found finished with every node cancelled and none ever claimed): - its conversation's snapshot is forgotten as seen and observed, and the graph is recorded per conversation in the scheduler state (graph_unclaimed), with the count of graphs let go in a row in that env; - no graph is admitted in that env until the next code version or the end of a hold of one hour, doubling with each let-go in a row up to a day, said once per hold (graph_admission_held, reason graph_unclaimed); - the next admission of that snapshot names the let-go graph in its key, so it is a new graph (and the same one again on a lost reply or a retried publication); every other admission keeps the key it always had; - a claim seen in that env resets the count (graph_unclaimed_cleared); - recovery no longer remembers as seen a snapshot whose graph had every node cancelled. The let-go line (graph_unclaimed) carries the reason, the count and the hold. The full-pipeline shadow goes on throughout; while held, the graph path makes no query. No SQL function, migration or Compose change. Tests, each seen failing on the previous code first: the let-go conversation is admitted again as a new graph after the hold, across a restart, with the full pipeline going on meanwhile; at the next build at once, with the hold doubling after a second let-go, and its given-up publication later found and only published; found cancelled or past the job deadline with no claim; a claim racing the cancel; the lines and their counts from the scheduler binary. And for the ten-minute rule: nothing is cancelled two seconds short of it; a graph whose claimed node was put back in the queue, and one with a node out of the queue and no attempt, are not let go; a re-admitted given-up publication leaves the given-up list and its nodes ran once. The held-admission test reads the scheduler's lines as they come instead of sleeping a fixed time. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * docs: a refused graph executor's blank list, the unclaimed hold and re-admission, the ten-minute constant, and what the copied-identity and publication-retry fixes do not cover The ten minutes are a constant, with what a slow box does. A refused executor starts the scheduler with its list blank. A graph let go as unclaimed holds admission for an hour, doubling to a day, or until the next code version, and its conversation is then admitted again as a new graph. Corrected: a copied identity still confirms an attempt whose original is alive but has stopped renewing for longer than a lease (writes stay fenced; the cost is a false exit record and the step computed twice); an honest reboot waits about one lease, up to a lease plus a reaper interval; a given-up publication is found and only published again only when the graph's specification is unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs * delphi shadow: the admission-key comment no longer names the admission function The one-admitter check refuses any file outside the shared entry that names the SQL admission function; a comment added with the let-go key named it. Comment only. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs --------- Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Contributor
Author
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release plan
Version: 2.45.3
Reason: no new selected migrations
Previous stable:
8289aae6781cfdcaa9d78f1b2fe5aa11af37dcc3Candidate:
99eb05482e9386ba9f1a39dfe05785dfadc8af73release.txt additions:
Merge the reviewed version preparation into edge, then refresh this plan for the resulting edge SHA before merging the promotion with a merge commit.