Emergency recovery: restore regulatory state and guard monitor mutations - #1158
Merged
Conversation
Restore the complete Federal Register and FINRA baselines from a current-main recovery run, keep primary and backup state identical, and preserve unrelated Learn state. Restrict mutating runs to the default branch and require captured base/state CAS before state PR creation or auto-merge; feature and PR refs remain dry validation only. Add regression coverage for the 494/57 versus 332/2 state-mixing failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class FetchResult(list): |
| result = fetch_page(request_url, session, max_retries=1) | ||
| try: | ||
| session._finra_last_request_at = time.monotonic() | ||
| except AttributeError: |
| if result['status_code'] not in (0, 429): | ||
| try: | ||
| session._finra_backoff_seconds = FINRA_RETRY_BASE_WAIT_SECONDS | ||
| except AttributeError: |
| FINRA_MAX_RETRY_WAIT_SECONDS, | ||
| max(FINRA_RETRY_BASE_WAIT_SECONDS, previous_wait * 2), | ||
| ) | ||
| except AttributeError: |
Keep feature and pull-request validation read-only by limiting state-change detection and default-branch CAS checks to successful default-branch monitor runs. Add regression coverage for the workflow conditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c90b12e50
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+86
to
+89
| wait_time = min( | ||
| MAX_RATE_LIMIT_WAIT, | ||
| max(retry_after_seconds, 2 ** attempt), | ||
| ) |
There was a problem hiding this comment.
When an endpoint returns 429 with a numeric Retry-After greater than 15 seconds, this cap retries before the server's requested window expires; with the default three attempts, a 60-second throttle is exhausted after roughly 30 seconds and returned as an error. This affects learn_monitor.py and change_radar_monitor.py, which call the shared fetch_page with default retries, causing legitimate throttling to produce noisy Learn reports or fail the change-radar run instead of recovering as the previous implementation did.
Useful? React with 👍 / 👎.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
judeper
force-pushed
the
recovery/regulatory-state-20260809
branch
from
August 9, 2026 11:15
60f8c82 to
f3d32e3
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| SCRIPTS_DIR = Path(__file__).resolve().parent | ||
| sys.path.insert(0, str(SCRIPTS_DIR)) | ||
|
|
||
| import monitoring_shared # noqa: E402 |
Require two-pass scoped listing proofs, preserve unresolved rows, and reconcile stable duplicates against authoritative detail. Regenerate coherent Federal Register and FINRA baseline state, report, and identical backup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recompute findings from the pre-incident watermark, require independent complete FINRA passes, and bind persisted identities to coverage proofs. Regenerate the coherent report and state while preserving Learn state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep Federal Register progress logs descriptive without emitting tainted date values. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace duplicate legacy FINRA entries with validated canonical identities and an alias ledger, preserving historical lookup compatibility while preventing stale aliases from satisfying coverage proofs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address the closeout review blockers on the regulatory recovery work. Workflow (`regulatory-monitoring.yml`): - The CAS step now publishes a `generated_manifest` recording every mutated path with both its git blob SHA and its sha256 content hash, and fails closed on an unexpected, missing or unhashable generated file. - Auto-merge consumes `state_sha_after`, the manifest, and the exact created head. It requires the PR to carry exactly the validated blobs, re-hashes the state on disk, binds the merge with `--match-head-commit`, and re-reads the live default-branch ref immediately before and after enabling auto-merge (revoking it if main moved). The historical `baseRefOid` is no longer the only base evidence. Every mismatch labels `needs-review`. Monitor (`regulatory_monitor.py`): - A `/node/<id>` URL used as a rate-limit transport fallback no longer replaces the canonical document identity, so an existing notice updates in place instead of being orphaned into an unevidenced alias migration. - Aliases are bound to verifiable facts outside the alias record: a migration may not move between two different FINRA notice numbers, and the alias must reach the hash the canonical entry actually carries via an explicit, contiguous, append-only content-update chain. Recomputing the ledger digest can no longer launder a redirected alias. - Every derived FINRA pass-proof count and digest is recomputed from the retained raw row payloads, and the duplicate ledger must account for coalesced rows, so replaced payloads cannot hide behind retained evidence. Tests: - Replace the frozen live-source totals (506/3616/3671/55/92/622) with named floors plus full internal reconciliation, digest recomputation and a complete `_validate_regulatory_state` assertion, so legitimate growth passes while truncation or regression still fails. - Add regressions for the FINRA 26-14 node fallback, forged alias redirects with a recomputed ledger digest, replaced pass payloads with retained digests, and the new workflow CAS/auto-merge bindings. State files are unchanged and remain byte-identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, forgery-resistant coverage Blocker 1 (workflow): remove unattended auto-merge from the privileged monitor workflow. No GitHub merge primitive binds BOTH the exact base and head atomically and there is no branch protection/ruleset, so auto-merge cannot be made fail-closed against a stale-main race. Stop at a validated PR for the existing external guarded sweep / human gate; drop every auto-merge success claim and the automerge_eligible signal. Blocker 2 (workflow): verify the created PR against the PR head commit's git blobs via the API instead of re-hashing the workspace (peter-evans create-pull-request restores the base checkout, so generated files are no longer on disk). Bind every CAS-manifested path to the exact blob at the PR head and require the PR to carry exactly the validated file set; fail closed (needs-review + exit 2) on any mismatch or missing binding. Blocker 3 (test): recovery test validates each snapshot strongly plus relationship/rollback semantics instead of requiring primary/backup byte equality (save_state_atomic intentionally stores the previous primary in the backup), so legitimate future state PRs are not blocked. Blocker 4 (monitor): resolve fetched FINRA identities through the trusted prior alias ledger before rebuilding entries, so a node-transport fallback that surfaces a migrated identity updates the existing canonical entry in place instead of orphaning it. Blocker 5 (monitor): bind pass-proof listing rows to the fetched entry set so a redirected alias with recomputed self-supplied evidence fails closed -- the repointed canonical, reachable only via that alias, drops out of the independent row reconstruction. Blocker 6 (monitor): validate duplicate-ledger record contents (recomputable raw_row_digest + a coalesced target that resolves into a fetched entry) and bind retained pass rows to fetched entries; reject forged retained rows in both passes and empty-dict duplicate records. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the Federal Register's official plain-text document URLs for generated reports so concurrent link validation avoids the flaky HTML frontend. Retry one complete link-check pass after transient failures while keeping 5xx responses fatal, and add regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use exact list equality in the workflow regression test so CodeQL does not misclassify list membership as incomplete URL substring sanitization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fail closed when refreshed FINRA details are not reconstructable from listing proofs, bind every legacy alias to its independently observed node, and document the verified-PR merge boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bind legacy FINRA aliases to retained detail-page evidence and a reviewed code-held recovery anchor. Fail closed on listing/detail divergence, document the verified-PR handoff, and add adversarial regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run destructive regulatory PR consolidation only after the created PR passes exact-head and generated-output verification. Correct the changelog and related guidance, and add ordering and output-gate regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve recovered Federal and FINRA coverage while retaining current-main Learn state and previous-main rollback chronology. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bind verified handoff and consolidation to the exact base, report same-day recovery gaps, and validate FINRA duplicate evidence against both retained passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fail closed unless retained duplicate listing evidence matches the hash-bound authoritative publication date, and validate persisted resolver evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
Owner batch disposition: withheld after final-guard rejection. The third independent revision at |
Owner
Author
|
PR hygiene mechanically updated the branch to current |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
🔍 Independent review by saul — NEEDS REVIEW (deferred for human merge):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Emergency recovery This PR repairs the regulatory-state regression introduced by maintenance PR #1157 without writing directly to
main. ### Verified incident At merged commite802babde, both regulatory entry maps remained at 332 Federal Register / 2 FINRA while the primary and backup watermarks advanced from 2026-07-20 to 2026-08-09. Neither snapshot carried a coverage proof, so the advanced watermarks could suppress the missing recovery window. ### Recovered state and report - Primary and backup each preserve identical regulatory recovery coverage: 506 Federal Register, 3,616 FINRA, and the same coverage proofs. They intentionally differ only in Learn rollback chronology: primary retains the current-main Learn state and backup retains the immediately preceding Learn state. - Federal Register coverage binds 170/170 documents, 2/2 pages, and the 2026-07-20 recovery window. - FINRA coverage binds 3,671 raw/resolved rows, 0 unresolved rows, 3,616 unique/detail identities, 92/92 pages, two matching independent pass proofs, and 622 aliases backed by retained raw detail-page identity envelopes. The envelope set has digestsha256:4277e4fbfb5548887f3c863e8f5e9979a75ae4563d8879025f66e5047d43e511; its canonical notice-to-node bindings match code-held anchorrecovery-2026-08-09-v1(sha256:4942b9dd838d4e893a1c07ecb140b70de96db7d57f5842e066e95a39195f1c89). - The unrelated Learn source state preserves currentmainin the primary snapshot and the immediately preceding Learn state in the backup snapshot. -reports/monitoring/regulatory-changes-2026-08-09.mdcontains 177 unique findings: 168 Federal Register + 9 FINRA, classified as 2 HIGH, 4 MEDIUM, 171 NOISE. Every report identity resolves in the recovered state. ### Link-check repair The failed check was not a dead-document failure: the Federal Register HTML frontend returned transient/random 500/503 responses (three links in CI and four different links in a local reproduction), while the same documents remained available through the authoritative API and plain-text endpoints. Generated reports now prefer the officialraw_text_url; the recovered report was rewritten only at those 168 link targets. The strict checker still rejects every 5xx response and retries one complete pass after a transient failure. No status code was allow-listed or ignored. ### Mutation guards - Feature and PR refs run offline dry validation only; mutation remains default-branch-only. - Missing/corrupt state, incomplete source coverage, identity conflicts, unexpected files, or undocumented exit codes fail closed before state PR creation. - A known FINRA detail refresh absent from both complete listing proofs makes the run incomplete; it cannot generate a report or save state. - Every legacy FINRA identity is bound to its specific canonical node by a retained raw detail-page envelope (canonical link, numeric shortlink,page-node-*body class, authoritative title) and the code-held recovery anchor. Mutable fallback URLs are transport-only and never accepted as identity evidence. Swapping aliases, fallback mappings, retained envelope node fields, hashes, and all mutable digests still fails validation. - After all in-memory source updates, the complete selected regulatory state is revalidated before any report generation or state save. - Default-branch and pre-run state SHAs are CAS-bound. - The created state PR must match the exact generated path/blob manifest through GitHub API read-back. - The privileged workflow does not auto-merge; it leaves the newest verified PR open for the guarded external/human merge gate. - Superseded-PR closure and branch deletion occur only afterverify_premitsverified=true; a failed or skipped verification leaves older trusted PRs untouched. - Verification requires the PR base SHA to equal the CAS-bound expected base and rechecks both base and head immediately before consolidation; any main movement fails closed. - Absent identities dated on the trusted watermark day are reported rather than silently baselined; unchanged known identities remain suppressed. - FINRA duplicate-ledger validation is exactly bound to both retained pass proofs, including occurrence count, target, node identity, payload, and detail hash. - FINRA duplicate date conflicts clear only when retained, normalized listing evidence matches the hash-bound authoritative detail date; conflicting, missing, ambiguous, or mutated resolution evidence fails closed. - Every duplicate node now requires independently hash-bound canonical node/date evidence tied to its persisted substantive-detail hash. Coherent retained-pass rewrites, resolver deletion, synthetic row fields, or wrong node/date/hash evidence fail closed even after mutable digests are recomputed.recovery-2026-08-09-v2(schema 1;sha256:48895b...ab1e4b), binding canonical URL, node, authoritative date, raw proof hash, and substantive-detail hash. Exact complete-set validation rejects coherent forgery even when all mutable state and digests are recomputed. ### Evidence replay and live limit - Current Federal Register recovery-window fetch: 170/170, 2/2 pages, all 170 hashes match the committed state, and all 170 records expose official raw-text URLs. - All 9 FINRA report details were fetched live and their substantive-content hashes match the committed state. - Offline reconstruction proves all 25/25 URLs in the current bounded FINRA refresh batch are represented by the retained listing proofs, and all 622/622 aliases are covered by retained detail envelopes whose full derived binding set matches the code-held anchor. - Read-only live detail retrieval established all 622 retained identity envelopes, but frequent FINRA 429 responses required bounded retries and numeric-node transport fallbacks; this is identity evidence, not a claim that a complete current archive replay succeeded. - A narrow live probe of FINRA notices 00-01 and 00-08 returned HTTP 200 and exposed the exact expected authoritative node shortlinks (/node/6547and/node/6540). - A full current FINRA two-pass listing replay was attempted read-only. It correctly failed closed because the live passes shifted under pagination/edge caching (3,671 rows in each pass; 3,614 vs 3,615 unique nodes) and produced repeated 429s. Therefore no new live FINRA watermark/state was persisted and no claim is made that a complete current archive replay succeeded. ### Validation - Regulatory recovery suite: 129 passed - Full scripts suite: 932 passed - Assessment suite: 483 passed - SPA suite: 221 passed, 1 skipped - Playwright: 15 passed - Ruff, strict MkDocs, 32 workflow parses, drift, link, SRI, duplicate-key, state/report consistency, andgit diff --check: passed ### Coupled PR disposition PR Monitoring: 0 CRITICAL, 0 HIGH regulatory items (144) #1162 was generated from the broken 332/2 baseline and carries no coverage proof. Its five Federal Register items and one FINRA item are already represented in this recovery; its hashes come from the legacy null-abstract/listing-only logic, and it misclassifies the FINRA detail as NOISE. Do not merge Monitoring: 0 CRITICAL, 0 HIGH regulatory items (144) #1162. After this PR lands, close Monitoring: 0 CRITICAL, 0 HIGH regulatory items (144) #1162 as superseded and run the repaired monitor from the newmain; create a replacement only if a fresh complete run produces independently verified changes. Recommendation: merge this PR only after the new head's required checks are green and through the guarded repository process.