Skip to content

Promote edge to stable: Polis 2.45.2 - #3450

Merged
colinmegill merged 4 commits into
stablefrom
edge
Nov 19, 2026
Merged

colinmegill merged 4 commits into
stablefrom
edge

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Release plan

Version: 2.45.2

Reason: no new selected migrations

Previous stable: c4f5d5b0bae985b0756fb73d829bfbfebff3f216

Candidate: 6cf1edfe7ee478bc0cbbcc42256b6eefe7dd5240

release.txt additions:

  • none

Merge the reviewed version preparation into edge, then refresh this plan for the resulting edge SHA before merging the promotion with a merge commit.

colinmegill and others added 4 commits November 18, 2026 03:15
…t the image can run, with one admission entry (#3446)

* delphi: graph steps and models come from one typed registry, checked against the image before admission

Which embedding, cluster, keyword and label models a job graph may name is
now defined once, in queue-rs/polis-contracts/job-models.json, read by a
typed Rust registry (Step, Model) and by the Delphi worker from a byte-equal
copy (a repository check compares the two files; a Rust test compares the
registry Python renders with the compiled one).

Admission goes through one Rust entry, jobs::admission, used by the shadow
scheduler and, as `polis-jobs graph-admit`, by the Python graph client and
the importer. Before the database call it asks the image which (step, model,
adapter digest) pairs it implements and refuses an unknown model, a model the
image does not carry, a wrong input role or producer, and out-of-range
resources. The worker child checks each input's producer schema again.

The placeholder models used by the graph proofs compile only into test
builds (the graph-test-models feature and the Delphi test image); the final
image and production binaries refuse them.

DELPHI_EMBED_MODEL_BY_ENV, optional, names the embedding model a result
environment uses. Unset, every path keeps its present model (granite at 384
dimensions); shipping weights for another model selects nothing.

No schema change: pd_graph_admit is unchanged and still checked last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* polis-contracts: job_graph.rs formats the same under both of its workspaces

The file is compiled in polis-contracts (the CLI workspace, edition 2021) and,
by path, in the queue-rs crate (edition 2024). The two rustfmt styles order
`use serde_json::{Value, json}` differently, so `cargo fmt --check` in the
CLI workspace refused what queue-rs's accepted. Import only `Value` and call
`serde_json::json!` by its path; no behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* ci: the one-admitter check reads the whole tree, not four directories

check_graph_registry.py looked for `pd_graph_admit` only under queue-rs/src,
delphi/scripts, delphi/polismath and server/src, so a new caller in
delphi/umap_narrative, the Delphi entry scripts, polis-api, the CLI or a shell
script passed. It now reads every tracked or unignored file (every file when
the root is not a Git checkout), skipping migrations and documentation, and
names the files that may mention the function: the Rust admission entry, the
database contract tests that call it directly, one documentation comment, and
the check itself. New tests cover callers in those places and the allowed
files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…r conversations' full-pipeline runs; optional same-host graph-stage daemon, off by default (#3447)

* delphi shadow: a job graph stuck past the job deadline is deferred and holds only its own conversation

When a graph's attempt cannot be proven finished (its scope stays held), the
scheduler no longer waits on it forever. After the job deadline the graph moves
to a deferred list that is polled one per cycle, and the full-pipeline shadow
keeps admitting every other conversation; only the conversation whose graph
scope is held is skipped. The environments the graph path has used are recorded
in the state file before any admission, so a graph admitted under a list that
was later emptied is still found and recovered after a restart.

The job daemon can bind its retained journal to one host identity file
(POLIS_JOBS_JOURNAL_HOST_FILE): a journal written under another host is refused
at start-up rather than read as exit evidence.

Adds Rust tests for a lost attempt after a restart with the list emptied and for
a committed admission whose reply was lost, and two Docker process checks for
recovery and for the scheduler under its deployed memory and CPU limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* delphi shadow: an optional graph-stage daemon with its own profile, journal volume and deploy check

Adds the delphi-shadow-graph Compose service (profile delphi-shadow-graph, off
unless DELPHI_SHADOW_GRAPH_ENABLED=1): the job daemon restricted to the four
graph stages at concurrency one, with its own journal volume, a read-only host
identity file bound in for the journal's host binding, and its own memory and
CPU ceilings (6 GiB and 2 CPUs by default). The daemon refuses to start unless
it runs as PID 1 of its own container with exactly those stages, concurrency
one and a host identity file set.

The deploy hook starts it only when the full-pipeline shadow is selected and the
setting is on, and first checks the rendered Compose settings with
scripts/check-shadow-graph.py (names only); if they are wrong, neither shadow is
started. With the setting off, nothing in the deploy changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* delphi shadow: graph publication is recognised by its served root and retried a bounded number of times; refusals are reconsidered by build

A finished graph's publication is now confirmed by reading the served bundle
and checking that its root artifact is this graph's leaf at the next
generation, not by the publish call's reply alone. A publish whose commit is
durable but whose reply was lost is recognised as published; a transient
database error before commit (connection, serialization, deadlock, lock or
too-many-connections classes) is retried at most three times, each on a fresh
connection, and then counted failed. A definite refusal is not retried. If the
served bundle cannot be read, the graph stays in flight and is resolved on a
later poll.

A refused conversation's recorded observation now carries the admission rule
and the build's code version, so a later build looks at it again once. A graph
admission that errored is followed by a recovery read, so an admission that
committed before its reply was lost is picked up before any full-pipeline run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* delphi shadow: graph wait and journal recovery log lines say why, and the recovery runbook is documented

A graph waiting past its deadline now logs the reason it is still held (nodes
still active, a process exit not yet confirmed, or reconciliation pending) and
the count of attempts with no confirmed exit, instead of one fixed reason. A
job daemon with a host-bound journal logs one line per retained journal entry
saying whether the attempt's exit was proven and why. docs/configuration.md
gains a section on the same-host graph executor: its profile, host identity
file, journal volume, deadlines, memory ceilings and what to drain before a
rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* delphi tests: the graph executor's Compose tests run in the Delphi test container

The graph executor's validator test located scripts/check-shadow-graph.py by
its own path, which inside the Delphi test container resolves to /scripts and
does not exist, so every shard of the Delphi Python tests stopped with a
collection error. It now reads the validator from the checkout the other
Compose tests read ($POLIS_CHECKOUT_DIR), and the deploy-hook runner copies the
validator there with its other inputs.

The test that the shadow yields CPU and memory to every other service failed
once the graph executor was added, because the executor carries the shadow's
own lowest weight. Both shadow daemons must now carry exactly that weight and
OOM adjustment, and every other service must still outrank them; no assertion
about any other service changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* delphi shadow: the graph executor receives the per-environment embedding model setting as the shadow daemon does

DELPHI_EMBED_MODEL_BY_ENV is now passed through to delphi-shadow-graph the way
the registry passes it to delphi-shadow and the scheduler: listed by name, with
no default of its own, so with the setting unset (today) it is absent from the
container and nothing changes. Each graph's model is still decided and sealed
by the scheduler at admission, and graph children's environment stays the
existing allowlist. The deploy check refuses a graph executor whose setting
differs from the scheduler's; tests cover the check and that both daemons list
the setting the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* graph registry check: the daemon recovery proof may admit its test node directly

delphi/tests/shadow/graph_recovery.py admits one test-model node through the
SQL admission function so that a test-build daemon has a real attempt to lose
and recover, as the graph proofs already do. The check that every product
admission goes through the shared entry now lists it beside those proofs. The
proof's description says it needs a test-build image, since a production
daemon refuses to run a test-only model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…y the comments participants are shown (#3448)

* shadow: tests that a run embeds every comment and lays out only what participants are shown

The Delphi shadow places, clusters and names every comment of a
conversation, rejected ones included. These tests say what it should do
instead, on comments in every moderation state (accepted, unmoderated,
rejected; inactive and muted ones among them; strict and open
conversations):

- the rule is the comment list participants are served
  (participantVisibleClause in server/src/comment.ts): a Rust contract test
  writes a golden of rows and what the rule keeps of them, open and strict,
  and the Python selection must keep the same comments under the same digest;
- a run that names the selection embeds every comment and gives UMAP, EVoC,
  the stored families and the naming only the selected ones; a run that
  names none (the served path) is what it was, and its pipeline arguments
  are byte for byte today's;
- the scheduler's census digests both sets, its admission names the
  selection and is keyed by it: a moderation that changes what participants
  are shown is a new run, one that does not (an unmoderated comment accepted
  outside strict moderation, a rejected comment added) is not; the floor
  counts the selection; a conversation an earlier build ran is run once more
  and then not again;
- the rehearsal sets moderation states on its public fixtures (one
  conversation under strict moderation; inactive, muted and unmoderated
  comments), counts its first wave by what participants are shown, adds two
  moderations that must run nothing, and checks the stored runs: every
  comment embedded, exactly the shown ones placed, clustered and named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* shadow: embed every comment once; lay out, cluster and name what participants are shown

Each Delphi shadow run now names its selection, participant_visible/1:
what participants are shown, by the rule of the comment list the server
gives them (server/src/comment.ts, participantVisibleClause: active,
velocity above 0, accepted under strict moderation or not rejected
otherwise), stated once in Rust (the input bundle contract's
participant_visible) and once in Python (polismath/selection.py).

- The pipeline, given --selection, embeds every comment with text and stores
  every embedding, then gives UMAP, EVoC, the keyword labels and every
  family derived from them (graph, assignments, keywords, features, names,
  the conversation's layout config) only the selected comments. The
  selection is the whole rule: the moderation and report filters must be
  off. Without --selection nothing changes: the served path, which the
  poller runs with no selection, embeds and lays out what it always did.
- The admission names the selection; the scheduler's census digests every
  comment (the embedded set, as the run's census records it) and the
  selection; the idempotency key is the selection's digest; the run record
  carries both digests and both counts (config_effective, delphi-shadow/2).
  The floor counts the selection, the cap and the storage estimate every
  comment with text. A state entry written before runs named a selection is
  read and admitted once more.
- The input bundle carries the conversation's strict_moderation and each
  comment's velocity only when the run names a selection; every other
  bundle is byte for byte what it was.

No schema change. No change to the vote, comment or next-comment path: the
census and the bundle read comments and conversations in read-only
snapshots that take ACCESS SHARE only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

* shadow rehearsal: embedded is every comment the newest run could read

A comment written after a conversation's newest run, and not shown to
participants (an unmoderated comment under strict moderation), starts no
run, so it is embedded by the conversation's next run, not before. The
rehearsal's selection check now holds the embeddings to every comment
with text the newest run could read, and counts the ones added since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor Author

Delphi Coverage Report

File Stmts Miss Cover
polismath/init.py 2 0 100%
polismath/benchmarks/bench_pca.py 128 107 16%
polismath/benchmarks/bench_repness.py 81 65 20%
polismath/benchmarks/bench_update_votes.py 38 28 26%
polismath/benchmarks/benchmark_utils.py 34 0 100%
polismath/components/init.py 1 0 100%
polismath/components/config.py 165 70 58%
polismath/conversation/init.py 2 0 100%
polismath/conversation/conversation.py 1293 264 80%
polismath/conversation/group_vote_counts.py 38 2 95%
polismath/conversation/manager.py 131 42 68%
polismath/database/init.py 1 0 100%
polismath/database/dynamodb.py 392 176 55%
polismath/database/postgres.py 414 95 77%
polismath/delphi_storage/init.py 1 0 100%
polismath/delphi_storage/codec.py 295 22 93%
polismath/delphi_storage/embedding_spool.py 58 6 90%
polismath/delphi_storage/golden_corpus.py 60 9 85%
polismath/delphi_storage/import_report.py 51 3 94%
polismath/delphi_storage/legacy_import.py 171 16 91%
polismath/delphi_storage/postgres.py 49 22 55%
polismath/delphi_storage/production_import.py 614 131 79%
polismath/delphi_storage/resource.py 179 32 82%
polismath/delphi_storage/scan_pacing.py 77 0 100%
polismath/delphi_storage/writer.py 196 13 93%
polismath/embedding_model.py 197 49 75%
polismath/empty_output.py 29 2 93%
polismath/engine_adapter.py 317 265 16%
polismath/job_child/init.py 364 51 86%
polismath/job_child/census.py 74 9 88%
polismath/job_child/input_bundle.py 106 7 93%
polismath/pca_kmeans_rep/init.py 5 0 100%
polismath/pca_kmeans_rep/clusters.py 257 38 85%
polismath/pca_kmeans_rep/corr.py 98 17 83%
polismath/pca_kmeans_rep/group_k_smoother.py 28 0 100%
polismath/pca_kmeans_rep/legacy_kmeans.py 197 8 96%
polismath/pca_kmeans_rep/pca.py 162 19 88%
polismath/pca_kmeans_rep/repness.py 226 5 98%
polismath/poller/init.py 4 0 100%
polismath/poller/admission.py 318 15 95%
polismath/poller/backfill.py 936 32 97%
polismath/poller/capacity.py 485 27 94%
polismath/poller/capacity_queue.py 248 13 95%
polismath/poller/coordinator_bridge.py 317 317 0%
polismath/poller/math_writer.py 120 3 98%
polismath/poller/promotion.py 112 4 96%
polismath/poller/readiness.py 368 13 96%
polismath/poller/rebuild_child.py 133 5 96%
polismath/poller/recompute_line.py 141 8 94%
polismath/poller/service.py 760 50 93%
polismath/poller/worker_pool.py 143 0 100%
polismath/queue/executor.py 221 111 50%
polismath/regression/init.py 4 0 100%
polismath/regression/clojure_comparer.py 188 20 89%
polismath/regression/comparer.py 891 640 28%
polismath/regression/datasets.py 126 27 79%
polismath/regression/recorder.py 36 27 25%
polismath/regression/utils.py 130 85 35%
polismath/replay/init.py 2 0 100%
polismath/replay/attribution_capture.py 49 0 100%
polismath/replay/axis_continuity.py 347 24 93%
polismath/replay/certify.py 999 59 94%
polismath/replay/crosslang.py 128 2 98%
polismath/replay/decision_ties.py 51 8 84%
polismath/replay/diagnostics.py 50 7 86%
polismath/replay/driver.py 73 1 99%
polismath/replay/dsn_admission.py 36 1 97%
polismath/replay/equiv_query.py 10 0 100%
polismath/replay/event_ingress.py 95 1 99%
polismath/replay/fixture_bundle.py 815 62 92%
polismath/replay/fixture_config.py 266 11 96%
polismath/replay/fixture_extract.py 285 12 96%
polismath/replay/fixture_generate.py 103 1 99%
polismath/replay/fixture_samples.py 157 14 91%
polismath/replay/fixture_selection.py 119 0 100%
polismath/replay/fixture_survey.py 147 11 93%
polismath/replay/legacy_pca.py 29 17 41%
polismath/replay/polarity.py 319 25 92%
polismath/replay/poller_equiv.py 776 99 87%
polismath/replay/prodclone.py 193 2 99%
polismath/replay/real_data.py 146 10 93%
polismath/replay/schedule.py 205 10 95%
polismath/replay/served_math.py 159 4 97%
polismath/replay/shard_bench.py 189 65 66%
polismath/replay/stagecompare.py 890 161 82%
polismath/replay/stages.py 256 40 84%
polismath/replay/stepcompare.py 86 1 99%
polismath/replay/store.py 120 11 91%
polismath/replay/tie_capture.py 107 19 82%
polismath/replay/types.py 58 6 90%
polismath/run_math_pipeline.py 284 93 67%
polismath/selection.py 48 0 100%
polismath/step_log.py 95 8 92%
polismath/tools/engine_rebuild.py 230 11 95%
polismath/types.py 6 0 100%
polismath/utils/cli_flags.py 15 0 100%
polismath/utils/clj_hash.py 44 0 100%
polismath/utils/env_flags.py 13 0 100%
polismath/utils/general.py 63 39 38%
polismath/utils/output_profile.py 66 2 97%
polismath/utils/serialization.py 10 5 50%
polismath/utils/vote_convention.py 159 5 97%
umap_narrative/700_datamapplot_for_layer.py 514 470 9%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 435 275 37%
umap_narrative/801_narrative_report_batch.py 657 209 68%
umap_narrative/803_check_batch_status.py 268 148 45%
umap_narrative/llm_factory_constructor/init.py 2 0 100%
umap_narrative/llm_factory_constructor/model_provider.py 232 106 54%
umap_narrative/narrative_data.py 178 55 69%
umap_narrative/numerical_stages.py 97 17 82%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 268 268 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 80 6 92%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 9 7 22%
umap_narrative/polismath_commentgraph/utils/converter.py 283 50 82%
umap_narrative/polismath_commentgraph/utils/group_data.py 332 279 16%
umap_narrative/polismath_commentgraph/utils/storage.py 616 486 21%
umap_narrative/reset_conversation.py 162 53 67%
umap_narrative/run_pipeline.py 459 158 66%
umap_narrative/topic_naming.py 148 11 93%
Total 26230 7392 72%

@colinmegill
colinmegill marked this pull request as ready for review November 19, 2026 04:54
@colinmegill
colinmegill merged commit 8289aae into stable Nov 19, 2026
102 checks passed

This branch was successfully deployed

1 active deployment
release — 6cf1edfe Deployed Nov 19, 2026 by colinmegill via publish #18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant