Skip to content

flip the served math to the Python engine: readers follow MATH_ENV, the Clojure writer keeps its prod label - #2911

Merged
colinmegill merged 1 commit into
edgefrom
pr-s2-flip-readers-only
Oct 2, 2026
Merged

colinmegill merged 1 commit into
edgefrom
pr-s2-flip-readers-only

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Switches the served math from the Clojure engine (label prod) to the Python engine (label python) by moving the readers only. Clojure keeps running and keeps writing prod for the rollback window; removing it is a later PR.

Mechanism

  • Readers follow the shared MATH_ENV in the production env secret (polis-web-app-env-vars): the server reads it from .env (Config.mathEnv), Delphi through ${MATH_ENV:-prod} in docker-compose.yml. Unchanged by this PR.
  • math-python writes python whatever the readers serve. Its label comes only from MATH_PYTHON_ENV (pinned python in the secret, compose fallback python). Its startup guard still refuses prod while MATH_POLLER_ALLOW_SERVED_ENV stays unset. Unchanged.
  • Clojure now writes ${MATH_ENV_CLOJURE:-prod}, not ${MATH_ENV:-prod}. Before this change, flipping the secret's MATH_ENV to python would have made Clojure write python beside math-python (two writers on one label). Now it keeps writing prod, so the rollback target stays current. Leave MATH_ENV_CLOJURE unset in the secret.
  • Dev: docker-compose.dev.yml gives Clojure ${MATH_ENV_CLOJURE:-${MATH_ENV:-prod}}, so existing dev .env files (MATH_ENV=dev) behave as before. docker-compose.test.yml is unchanged (its own math service, dev).
  • Nothing else in compose or scripts/after_install.sh derives Clojure's label from MATH_ENV. The unknown-role fallback in the hook (up -d of every unprofiled service) now also starts Clojure on prod, not on the served label.

Operator sequence

  1. Merge to edge, then to stable. Boxes take docker-compose.yml from stable at deploy, so this must land before the secret changes; under an older compose file Clojure follows MATH_ENV. Do not move stable back past this change while the secret says python.
  2. Confirm every servable conversation has a python row (the backfill covers conversations older than the poller's boot window).
  3. Set MATH_ENV=python in the env secret.
  4. Deploy. The hook rewrites .env from the secret on every box; server and Delphi restart reading python; the math box restarts Clojure, still on prod.
  5. Verify: /api/v3/math/pca2 for an active conversation returns an ETag starting "python- (was "prod-) with groups and base clusters present; Delphi reports find their math rows; max(math_tick) under prod keeps advancing.

Clients refetch after the flip without further work: pca2's "<math_env>-<math_tick>" ETag (already on edge) is compared as an opaque value, so a browser holding a prod- tag gets a 200 with the python body, even at an equal tick.

Rollback: set MATH_ENV=prod in the secret and redeploy. No hook or compose edit.

Changes

  • docker-compose.yml: math reads MATH_ENV_CLOJURE; comments on delphi and math-python name the label each reads or writes.
  • docker-compose.dev.yml: dev fallback described above.
  • example.env, math/example.env, docs/configuration.md: document MATH_ENV_CLOJURE ("the label the legacy Clojure engine writes; keep prod").
  • scripts/after_install.sh: comments only. No role starts or stops anything new.
  • delphi/docs/CUTOVER_RUNBOOK.md: "Step 2 as built" (flip, verification, ETag rule, rollback).
  • delphi/tests/test_compose_math_env.py: Clojure writes prod under any MATH_ENV (unset, prod, python, a probe value, empty MATH_ENV_CLOJURE); its label comes only from MATH_ENV_CLOJURE; the switch and rollback move only Delphi; the two writers never share a label. The production-stack agreement test is now scoped to the test stack, where reader and writer still share MATH_ENV. Four of the new cases fail on the previous compose file.

What was dropped from #2895 and why

This replaces the draft #2895 (based on an older edge). Nothing from it is cherry-picked.

  • Server hunks (pca2 ETag, isSynthesizedPca, tests): already on edge (75be08a35, 7d452d41c). git diff origin/edge origin/pr-s2-math-python-flip -- server/ shows only edge being ahead, so no server hunk is needed to read python rows.
  • after_install.sh math role starting nothing / starting Clojure only under prod with a retirement check, and the fail-closed unknown role: dropped. Clojure keeps running through the rollback window; with its label decoupled there is nothing to gate.
  • math/bin/run label guard and its shell test, and the python-ci.yml copy step: dropped with the above; the compose decoupling removes the case it guarded against.
  • Deploy workflow changes (write-once S3 object per run, ETag/version binding, hash checks): dropped; independent of the flip and can return as their own PR.
  • Runbook text describing Clojure's retirement in step 2: replaced by the readers-only description.

Tests

  • delphi: pytest tests/test_compose_math_env.py tests/test_before_install_hook.py: 37 passed.
  • server: jest __tests__/unit/pcaMathTickZeroHttp.test.ts __tests__/unit/pcaMathTickZero.test.ts: 50 passed (no server change; run to confirm the label-prefixed ETag on this base).
  • docker compose config resolves math/delphi/math-python to prod/prod/python with MATH_ENV unset or prod, and to prod/python/python with MATH_ENV=python; with the dev overlay and MATH_ENV=dev, Clojure and Delphi both resolve to dev.

No schema change.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

…he Clojure writer keeps its prod label

The served-label switch moves only the readers. The server (MATH_ENV from
.env) and Delphi (${MATH_ENV:-prod} in compose) follow the shared MATH_ENV
in the production env secret; setting it to `python` serves the rows
math-python has been writing, and setting it back to `prod` is the rollback.

Before this change the Clojure `math` service also took ${MATH_ENV:-prod},
so the switch would have made Clojure write `python` beside math-python: two
writers on one label. Clojure now writes ${MATH_ENV_CLOJURE:-prod}, so it
keeps running and keeps its `prod` rows current through the switch, which
is what a rollback serves. math-python's label still comes only from
MATH_PYTHON_ENV (default `python`). The dev overlay keeps Clojure on
MATH_ENV unless MATH_ENV_CLOJURE is set, so existing dev .env files behave
as before.

- docker-compose.yml: `math` reads MATH_ENV_CLOJURE; comments on delphi and
  math-python say which label each reads or writes.
- docker-compose.dev.yml: `math` falls back to MATH_ENV in dev.
- example.env, math/example.env, docs/configuration.md: document
  MATH_ENV_CLOJURE ("the label the legacy Clojure engine writes; keep prod").
- scripts/after_install.sh: comments only; no role starts or stops anything
  new. Clojure keeps running on the math role.
- delphi/docs/CUTOVER_RUNBOOK.md: "Step 2 as built": merge to stable, secret
  MATH_ENV=python, deploy, verify the pca2 ETag prefix is `python-`;
  rollback MATH_ENV=prod and redeploy; retiring Clojure is a later PR.
- delphi/tests/test_compose_math_env.py: Clojure writes `prod` whatever
  MATH_ENV says, its label comes only from MATH_ENV_CLOJURE, and the switch
  and rollback move only delphi. Four of these fail on the previous compose.

No server change (the label-prefixed pca2 ETag is already on edge). No
schema change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1290 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 178 55%
database/postgres.py 324 100 69%
empty_output.py 29 2 93%
engine_adapter.py 316 264 16%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 8 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 927 32 97%
poller/coordinator_bridge.py 316 316 0%
poller/math_writer.py 113 3 97%
poller/readiness.py 341 10 97%
poller/service.py 506 33 93%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 283 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 777 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 264 114 57%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 112 53 53%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 434 275 37%
umap_narrative/801_narrative_report_batch.py 790 726 8%
umap_narrative/803_check_batch_status.py 185 185 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 234 234 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 331 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 584 470 20%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 374 230 38%
umap_narrative/topic_naming.py 148 148 0%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 29 2 93%
Total 21698 8166 62%

@colinmegill
colinmegill merged commit 5788918 into edge Oct 2, 2026
5 of 6 checks passed
@colinmegill
colinmegill deleted the pr-s2-flip-readers-only branch October 2, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant