Repository navigation
SCHEMA CHANGE (draft, not for merge without a ruling): the Polis 2 permission can be recorded in Postgres, one new table, switched off - #3451
Draft
colinmegill wants to merge 5 commits into
Draft
colinmegill wants to merge 5 commits into
colinmegill wants to merge 5 commits into
Conversation
… accounts may use Polis 2 (not yet read)
One table and nothing else: no existing table, column, index, constraint,
function, trigger, rule or grant changes, and the file creates no function,
trigger or rule.
delphi_entitlements
uid integer primary key, references users(uid) on delete cascade
granted_at bigint not null, default now_as_millis()
granted_by integer, references users(uid) on delete set null
source text not null, check in ('auth0_role', 'site_admin')
Nothing reads or writes it in this commit. The server reads it only under a
setting that is off by default, added in the next commit.
Locks, measured on PostgreSQL 17.11: the two foreign keys take SHARE ROW
EXCLUSIVE on users until commit, 3 ms median and 6 ms at most over 20 runs.
While it is held, reads of users, inserts into participants and writes to
conversations went on without waiting; inserts and updates of users (a new
anonymous participant's account, a first sign-in) waited for the commit.
Beside an open transaction writing users the file waits its 5 s lock_timeout
and fails with nothing changed, once, at the deploy. Nothing on the vote,
comment or next-comment path reads the table.
Declared additive (compat.txt, with its evidence in docs/migrations.md); listed
in release.txt; the forward proof's additive ledger and the first-deploy
report's release list include it, and the report is regenerated. The
reversal, down/000048_drop_delphi_entitlements.sql, refuses while any row is
stored; dropping the foreign keys takes ACCESS EXCLUSIVE on users until
commit (milliseconds, 5 s at most).
The coordinator schema census follows the new table: catalog.json measured
by coordinator-rs/tools/schema_census.py on a fresh PostgreSQL 17.11 (the new
table, its index and four constraints, and the file's checksum); the row type
and table entry in rows.rs; 770 columns (was 766) in the census test, the
mapping example and the README. The characterization schema catalog
(server/characterization/catalog.json, read only when recording) is refreshed
in its own change, as the recording rules require.
Test: an integration test against the migrated test database checks the four
columns, types, nullability and default; no trigger and no rule; a grant from
each source; one row per account; a source outside the closed set, a missing
source or time, and an absent account or granter refused; deleting an
account deletes its row, deleting its granter keeps the row and clears
granted_by.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…nd POLIS_DELPHI_ENTITLEMENT (default claim: unchanged); GET /api/v3/delphi/me answers it
isDelphiEntitled (server/src/routes/delphi/entitlement.ts) answers whether an
account may use Polis 2. The four routes that start Delphi work ask it first,
in place of reading req.p.delphiEnabled themselves:
POST /api/v3/delphi/jobs (routes/delphi/jobs.ts)
POST /api/v3/delphi/batchReports (routes/delphi/batchReports.ts)
POST /api/v3/collectiveStatement (routes/collectiveStatement.ts)
GET /api/v3/reportNarrative (routes/reportNarrative.ts)
and then apply the conversation-owner rule (generationOwner.ts) exactly as
before. A refusal is the reply each route already gave a caller without the
claim, byte for byte.
POLIS_DELPHI_ENTITLEMENT (config.ts, delphiEntitlement) says where the answer
comes from:
- claim (unset or blank: the default): the token's delphi_enabled claim, as
these routes always read it. The database is never read for this.
- claim_or_db: the claim, or else a row for the account in
delphi_entitlements (migration 000048).
- db: a row only.
Any other value stops the server at startup.
Under claim_or_db and db the read is one SELECT by primary key
(SELECT 1 FROM delphi_entitlements WHERE uid = ($1);) on the ordinary
read-write pool, on these routes only; it holds its connection for that
statement and nothing else. These routes never run in participation scope,
so votes, comments and the next comment never make it and cannot wait on it.
A failed read answers through each route's existing error reply and nothing
starts.
New route: GET /api/v3/delphi/me, signed-in callers only (hybridAuth: 401
without a token, as every such route), answers {"delphi_entitled": true|false}
through the same function. Its path is made of words the client error
reporters already know, so no client copy changes; no nginx Rust selector
matches it (the /api/v3/delphi selectors are exact paths, and the one pattern
is /api/v3/delphi/runs/<id>). The route census gains it (id 211).
Not moved: the vote CSV import, POST /api/v3/votes-bulk (routes/votes.ts),
still reads req.p.delphiEnabled directly under every value of the setting. A
change to that handler is proposed separately.
Documented in docs/configuration.md ("Polis 2 permission") and example.env.
Tests (written first; on edge the new suites fail and the import's passes):
- unit, delphiEntitlement.test.ts: the function under each value (no read of
any kind under claim, whatever the claim; one read by primary key on the
read-write pool otherwise; no read for an account number that is not one; a
failed read throws; an unknown value answers as claim); the four routes
under each value (under claim, the owner with the claim proceeds after only
the owner read, and an account with a row but no claim gets today's
refusal with nothing read; under claim_or_db and db, a row admits its owner
and does not reach another's conversation, the claim alone is refused under
db, a failed read starts nothing); the new route under each value.
- unit, delphiEntitlementSetting.test.ts: blank and unset are claim; the three
values are taken as written; anything else stops the server.
- unit, votesBulkEntitlementUnchanged.test.ts: under every value the import
refuses an account without the claim and admits a claim holder to its owner
check, reading no entitlement.
- integration, delphi-entitlement.test.ts, against the real stack: 401
without a token; under claim the claim decides and a row is ignored; under
claim_or_db either one; under db the row decides, and an owner with a row
starts a pipeline run while the same owner without it gets today's 403.
The existing tests of these routes (delphiGenerationOwner, votesBulkOwner,
the generation-owner and claim integration tests) are unchanged and pass.
- The collective-statement mutation check: the mutation that drops the
permission check in collectiveStatement.ts now finds the new line (its text
moved; the same five cases fail), and a new mutation, ENTITLEMENT-DEFAULT,
makes the function's default answer yes for everyone and fails the same
five cases.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
… claim Profile (queue-rs/polis-auth/src/oidc.rs) mapped the namespaced delphi_enabled claim as request metadata. Nothing read it: no polis-api route calls Profile, and the only callers are this crate's own tests. It is removed, so no Rust route can be written that reads the claim; whether an account may use Polis 2 is recorded by Polis (the delphi_entitlements table), and a Rust route that needs it reads it there. A token that carries the claim is mapped exactly as before, minus that field. Test (written first; it failed on the previous source, naming oidc.rs): no source file of polis-auth, and none of polis-api, contains the claim's name. The claim-fallback test still feeds the claim and no longer reads it back. The README says the claim is not mapped. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
Brings in edge up to the 2.45.2 preparation: the Delphi shadow's comment selection. No conflicts; no file of this branch changed on edge. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…runner's 5 s, so no vote or comment is refused while it waits Creating delphi_entitlements takes SHARE ROW EXCLUSIVE on users (its two foreign keys). While the migration waits for that lock, behind a session holding a conflicting lock on users (a transaction that wrote users, or a vacuum of it), every new writer of users queues behind it. A signed-in account's first request creates its users row in a transaction that gives up on a lock after 500 ms (server/src/db/pg-query.ts) and is then answered HTTP 500, and that request can be a vote or a comment. With the runner's 5 s, such a first vote sent during the wait was refused. The file now sets lock_timeout to 200 ms. On a busy instant the migration gives up after about 0.2 s with nothing changed (no table, no history row), the deploy stops on that box before any service is replaced, and it is run again. Nobody waits longer than that and nobody is refused. The table and both foreign keys are unchanged. The reversal (down/000048_drop_delphi_entitlements.sql), which takes ACCESS EXCLUSIVE on users, gets the same bound; a refused down changes nothing. The file header and its declaration in docs/migrations.md now say what it locks, for how long and why 200 ms (they said nothing was refused), and that the foreign keys add PostgreSQL's internal referential triggers. The coordinator schema census follows the new bytes (the checksum, re-measured on a fresh PostgreSQL 17.11, and the declaration pointers, seven lines lower), as does the first-deploy report. Test: server/__tests__/participation-pg/delphi-entitlements-migration.pgtest.ts, run by scripts/test-delphi-result-reads.sh. An open transaction writes users; the real runner's deploy waits on it; while it waits, a signed-in account's first vote, two anonymous first votes and, once they have queued, an existing participant's vote run the middleware of POST /api/v3/votes in order (onParticipationPools, the token step, ensureParticipant, handle_POST_votes), with the participation pools at 3 connections. Without the file's own bound (the runner's 5 s) the signed-in first vote is answered 500 and not recorded. With 200 ms every vote is answered and recorded, the migration fails with nothing changed, and the next deploy applies it. The reversal, under the same bound, gives up with nothing changed beside a transaction reading users, and drops the empty table once it can. Run against the 5 s file, the released case and the reversal case failed; with 200 ms all three pass. Measured on PostgreSQL 17.11, 20 runs with each file: with 200 ms every vote was answered, the longest after 0.27 s; with 5 s the signed-in first vote was answered 500 in all 20, and the anonymous first votes waited 5 s. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
Contributor
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SCHEMA CHANGE: one new table
The Polis 2 permission can be recorded in Postgres: one new table, one server function behind a setting that is off, and a read route for the signed-in caller
Observable changes
delphi_entitlements(migration000048), created at the next deploy. Nothing reads or writes it with the default setting. Its creation holds a lock onusersfor a few milliseconds, once; ifusersis being written at that instant it waits at most 0.2 s and then gives up, failing the deploy cleanly (below).GET /api/v3/delphi/me. A signed-in caller gets{"delphi_entitled": true}or{"delphi_entitled": false}; without a token, 401{"error":"No authentication token found"}, as every signed-in route answers. With the default setting the answer is the access token'sdelphi_enabledclaim, so it says exactly what the Polis 2 routes would decide before their owner check. No client calls it yet.POLIS_DELPHI_ENTITLEMENT, defaultclaim. Unset or blank isclaim. Any value other thanclaim,claim_or_dbordbstops the server at startup.Nothing else changes with the default. Every reply of the four Polis 2 routes (
POST /api/v3/delphi/jobs,POST /api/v3/delphi/batchReports,POST /api/v3/collectiveStatement,GET /api/v3/reportNarrative) is byte for byte what it was, and the database is never read for the permission. The collective-statement and Delphi recordings replay unchanged.The table
One row per account that may use Polis 2: when it was granted, who granted it, and whether it was carried over from the identity provider's role or granted by a site administrator. Deleting an account deletes its row; deleting the granter keeps the row and clears
granted_by. No existing table, column, index, constraint, function, rule or grant changes, and the migration creates no function, trigger or rule of its own (the two foreign keys add PostgreSQL's internal referential triggers onusersand on the new table). Declared additive: a release without it runs beside it unchanged.Locks at the deploy (measured on PostgreSQL 17.11): the two foreign keys take SHARE ROW EXCLUSIVE on
usersuntil the migration commits, 5 ms median and 9 ms at most over 40 runs. Reads ofusersdo not wait for that lock, and neither do votes, comments and the next comment from people who already have an account. Creating ausersrow does: a new anonymous voter's first vote or comment, and a first sign-in.Who can wait, and for how long. If another session holds a conflicting lock on
usersat the instant the migration starts (a transaction that has writtenusersand not yet committed, such as an import or another first sign-in, or a vacuum ofusers), the migration waits for its lock and every newusersrow queues behind it. The migration's own lock timeout is 0.2 s, not the migration runner's 5 s, because a first sign-in gives up on a lock after 0.5 s and is answered HTTP 500, and that request can be a vote or a comment. So the longest anyone waits is about 0.2 s: new accounts (a new anonymous voter's first vote or comment, a first sign-in) and, because each waiting first vote holds one of the box's vote connections, briefly other votes on that box. Nobody is refused. Measured 20 times beside an open transaction writingusers, with three new accounts' first votes queued behind the migration (one signed in, two anonymous) and an existing participant's vote waiting for a vote connection: the migration gave up within 0.33 s (it waits 0.2 s; the rest is the runner starting), every vote was answered (the test also checks that each is stored), and the longest wait was 0.27 s (signed-in first votes 0.22 to 0.27 s, anonymous first votes 0.22 to 0.27 s, the existing participant's votes 0.20 to 0.25 s). With the runner's 5 s instead, the signed-in account's first vote was answered HTTP 500 in all 20 runs, the anonymous first votes waited 5.0 s, and the existing participant's vote 0.5 s.On a busy instant the migration gives up after 0.2 s with nothing changed: no table and no history row. Nothing on that box has been stopped yet: the hooks before it stop no service, and the migration runs before any service is replaced (
scripts/after_install.sh, which stops on the failure). The box was already taken out of the load balancer, so the deploy fails there and goes no further, and the other web boxes keep serving the previous release. CodeDeploy's automatic rollback on a failed deployment (on for both deployment groups) then redeploys the previous release, one box at a time, like any deploy. The deploy is then run again.Nothing on the vote, comment or next-comment path ever reads the table.
Reversal:
down/000048_drop_delphi_entitlements.sqldrops the table only while it is empty and is never run automatically. It takes ACCESS EXCLUSIVE onusersfor milliseconds, under the same 0.2 s bound: whileusersis busy it gives up with nothing changed (the table, its rows and the history row stay) and is run again later.The setting
POLIS_DELPHI_ENTITLEMENTclaim(default)delphi_enabledclaim, as todayclaim_or_dbdelphi_entitlementsdbThe read is
SELECT 1 FROM delphi_entitlements WHERE uid = ($1);on the server's ordinary read-write pool, on the four Polis 2 routes and the new route only. Those routes never run on the participation pools, so votes, comments and the next comment never make the read and cannot wait on it; the read holds its connection for that one statement. A failed read answers through each route's existing error reply, and nothing starts. The owner rule after it (the conversation's owner or a site administrator) is unchanged in every mode.Not moved
The vote CSV import,
POST /api/v3/votes-bulk, still reads the claim directly under every value of the setting. Changing that handler is proposed separately. Until it is, an account recorded only in the table can start Delphi work but not import votes, and underdba claim holder can still import into its own conversation.Rust
polis-auth's OIDC profile no longer carries the namespaceddelphi_enabledclaim. Nothing read it (no polis-api route uses the profile); a Rust test now checks that no source file of polis-auth or polis-api names the claim.Commits
release.txt,compat.txt(additive),docs/migrations.md(the declaration and its lock measurement), the forward-compatibility ledger, the regenerated first-deploy report; the reversal; the coordinator schema census (measured on PostgreSQL 17.11) follows the new table; an integration test of the table's constraints. The characterization schema catalog, which only the recorder reads, is refreshed in its own change after this one, as the recording rules require.isDelphiEntitled(server/src/routes/delphi/entitlement.ts) and the four routes asking it; the setting (server/src/config.ts);GET /api/v3/delphi/me; docs (docs/configuration.md, "Polis 2 permission") andexample.env; the route census; unit tests of the function, the routes and the setting under every value, a test that the vote import is unchanged under every value, and an integration test of the route and the setting against the real stack. The collective-statement mutation check follows the moved line and gains a mutation of the function's default; both are caught by the recordings.docs/migrations.md, the schema census (checksum and line pointers) and the first-deploy report follow. A test,server/__tests__/participation-pg/delphi-entitlements-migration.pgtest.ts, holds a transaction writingusers, starts the deploy with the real migration runner, and while it waits sends a signed-in account's first vote, anonymous first votes and an existing participant's vote through the server's own middleware forPOST /api/v3/votes.Tests
Written before the code; on edge the new server suites fail (the vote-import one pins today's behaviour and passes) and the Rust test fails naming the file that mapped the claim.
Run locally, on the work machines: every check these paths trigger (38), including the migration runner checks (fresh image: 38 receipts), the coordinator schema census, the server unit and integration suites (191 suites, 3144 tests), the route census, every recording replay (collective statements with their mutation check, Delphi characterization, identity, post-comments, treevite, routing), the CLI differential, the API checks (1351 Rust tests), the queue and authority checks, the vote-sign lint and the six end-to-end shards. One end-to-end test that also fails intermittently on other branches (the alpha client's anonymous-session reload) failed once and passed twice when run again; the result-reads check failed once while its database was starting and passed when run again.
The lock timeout has its own test, written first: run against the 5 s file it failed (the signed-in account's first vote was answered 500, and the reversal waited 5 s); with 0.2 s it passes. After that change these ran again on the work machines: the migration runner checks (fresh image: 38 receipts), the coordinator schema census test, the result-reads and participation suites where the new test lives (17 suites, 97 tests) and the server unit and integration suites (191 suites, 3144 tests). The full set above was not run again for a changed timeout, its comments and a test.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
🤖 Generated with Claude Code
https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs