Repository navigation
feat(verify): durable verification reports that outlive Temporal retention, with linked reconstruction and re-verification - #163
Open
NovrusShehaj wants to merge 23 commits into
Conversation
- Updated `compose.yaml` to configure Temporal with a persistent SQLite database. - Added new settings in `config.py` for Temporal namespace and verification report collections. - Implemented verification report persistence activities in `activities.py` and integrated them into workflows. - Enhanced error handling and logging for report persistence in `main.py` and `runs_verify_workflow.py`. - Introduced new models for report persistence state and artifact references in `models.py`. - Updated OpenAPI specifications to reflect new report persistence features. This commit improves the reliability and traceability of verification reports in the system.
…y support
- Added new endpoints `/verify/{workflow_id}/reconstruct` and `/verify/{workflow_id}/reverify` to handle reconstruction and reverification of simulations.
- Introduced models for `ReconstructRequest`, `ReverifyRequest`, and `LinkedRecoveryAccepted` to support new workflows.
- Enhanced the `VerificationDatabaseService` with methods for managing recovery reservations and derived verifications.
- Updated OpenAPI specifications to include new endpoints and their parameters.
- Improved error handling for idempotency conflicts and recovery scenarios in the verification process.
This commit enhances the verification system's capabilities for handling complex recovery scenarios while ensuring idempotency and traceability.
- Updated the `/verify/{workflow_id}` endpoint to include `derived_verifications` and `recovery` offers in the response when the original report is unavailable.
- Modified the `_ResultsNoLongerRetained` exception to include derived verification links.
- Enhanced the GCS storage handling to map HTTP 404 errors to `FileNotFoundError` for missing objects.
- Updated frontend components to display derived verifications alongside original results, ensuring clarity in the verification status.
- Improved test coverage for derived verifications and recovery scenarios.
This commit improves the user experience by providing additional context for verification results and enhances the robustness of the verification system.
The CronJob lived in kustomize/base, which every overlay includes. Merging to main would have made Flux create it in biosim-gke on backend-0.10.3, which lacks scripts.reconcile_verification_reports, so it would fail every five minutes until a release containing the module was deployed. It is now kustomize/components/verification-reconcile. No overlay enables it; the deploy PR that moves an overlay to a release with the module adds the component. All four overlays render byte-identical to the PR base. The job also mounted worker-cache-pvc, which is ReadWriteOnce and held by the worker, so a pod on another node could never attach it. /app/scratch only holds one run's temporary downloads, so it is now an emptyDir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inistic A verification report must never be a degraded version of a computed result, and two writers that recover the same execution must produce byte-identical content. Both properties were broken in three ways: * The GET path and the reconciler each synthesized their own FAILED report. The GET wrote "Verification workflow ended with status X" with the execution's close time; the reconciler wrote "Temporal execution ended with X" with the ledger row's creation time. The first writer to insert won, so the second one hit a digest conflict and answered 500. * Neither writer looked at the workflow history. When a workflow computed an output and then failed to store it, the computed output was still in the history as the persist activity's input, and both writers threw it away. * report_recovery imported api.main to read the start event, so the reconciler CronJob loaded FastAPI. The history helpers now live in biosim_verify/history.py: read_start_input decodes the start event, computed_output_from_history returns the input of the last scheduled persist activity, and terminal_failure_output is the one deterministic FAILED builder. Both writers use them, so they agree on the content and on the bytes. The computed output wins: a terminal failure is synthesized only when the history proves nothing was ever computed, so a successful verification can never be replaced by a failure. The persist activity now returns the slim result once its report is durable: identity, status, error, failed_simulators, compare_settings, timestamps and the STORED state stay, while workflow_results and workflow_results_reference are cleared. The stored report is authoritative, so statistics leave the activity result, the workflow result and every query. A BLOCKED result keeps them, because that is the only copy of the truth. This is what makes test_real_temporal_and_mongo_roundtrip[completed] pass: the assertion and the activity return belong to the same change. A losing writer now serves the stored report instead of failing the request, and a STORED result whose report is missing is an incident (503 with an attention log), never a slim result dressed up as the report.
The persist activity call was duplicated in each verification workflow, and every copy gave up after 20 attempts with a 1-minute maximum interval. A storage outage longer than that turned a computed result into a permanent failure: the workflow completed FAILED even though the science had succeeded. workflow_persistence.py now holds the single call site and the single policy. Transient failures retry without a cap, so a computed result is never degraded because storage was briefly unavailable. A permanent failure (a ValueError, a report integrity conflict, a corrupt artifact, or a missing artifact) completes the workflow with the computed output marked BLOCKED: the verification itself succeeded, and a workflow that keeps retrying also keeps its history, which is what recovery reads. The activity now classifies its own failures. ReportArtifactMissing is permanent and is listed explicitly because it subclasses ReportUnavailable, which stays retryable: an unconfigured or unreachable store may recover, but a missing object never appears later. A ledger read failure is raised as retryable instead of being swallowed, so a report is never stored with its re-verification link silently dropped. The capped_retry_runs fixture records a history under the old 20-attempt policy, so the replay test proves in-flight and finished executions from the previous build still replay under this one.
Statistics crossed Temporal in three places: the statistics activity result, the persist activity result, and the workflow result and query. A large comparison could therefore exceed the payload limit even after its report was safely stored. generate_statistics_activity_v2 now returns statistics inline when their encoded JSON is at most verification_activity_inline_max_bytes (512 KiB, a quarter of Temporal's 2 MiB limit) and externalizes them otherwise. Most verifications then skip an object write and read and create no transient artifact, and history recovery can read inline statistics even when object storage is unavailable. test_workflow_payloads.py decodes the persist ActivityTaskCompleted result, the workflow result and the final query on a real worker and asserts none of them carries workflow_results, that the stored report still does, and that the inline threshold is inclusive at its boundary.
Deploying this branch replaces the worker that reads histories written by 0.10.3. If the new converter cannot decode an old payload, every retained history becomes unreadable at exactly the moment the drain procedure is supposed to be safe. test_converter_compat.py round-trips all 16 payload samples (every 0.10.3 workflow and activity type, with NaN, +/-inf, enums, datetimes and None) from DataConverter.default to pydantic_data_converter, and decodes a history written with the default converter through read_start_input, result() and the Replayer under the current converter. No decoding shim is needed, which is recorded here rather than assumed.
The reconcile job exited 1 whenever a row needed attention, and its CronJob retried a failed run in place. One blocked report therefore failed every scheduled run until someone intervened, and a failing job was indistinguishable from a healthy job that had found work. The exit status now reflects the job's health only. A scan that ran, complete or budget-limited, exits 0 and always logs verification_reconcile_summary as the heartbeat; it adds verification_persistence_attention (ERROR) only when a row needs an operator. A lease held by another run exits 0 with skipped="lease_busy". Initialization failure, a lost lease and unexpected exceptions exit 1 with verification_reconcile_job_failed, which carries the reason and the exception type but never its message. Staleness now comes from the execution's pending activities instead of the ledger row's age. A persist activity at attempt 3 or later is persist_retrying and needs attention after --stale-persist-after (1h); its age is a lower bound derived from the retry policy, because Temporal does not report when the first attempt was scheduled. Anything else running is computing and needs attention only after --stale-running-after (24h), since a comparison can legitimately take that long. The CronJob uses restartPolicy: Never and backoffLimit: 0, so the next 5-minute run is the retry. The heartbeat metric counts verification_reconcile_summary, and the attention alert matches verification_persistence_attention or verification_persistence_failure. Every overlay still renders byte-identical to the base branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row that needs an operator, such as a report blocked by an encoding bug,
alerted on every 5-minute run until the report was stored. The only way to
silence it was to change the data, which the immutability guarantee forbids.
The ledger row gains persistence_ack = {state, error, reason,
acknowledged_by, acknowledged_at}. It applies only while the row's
persistence_state and persistence_error still equal the acknowledged values,
so a different failure brings attention back. The acknowledgement never
deletes or changes a report.
scripts/ack_verification_persistence.py writes it. It is dry-run by
default and prints the row's persistence fields (never its owner or saved
request). It refuses stored rows, rows that are not in --expect-state, and
rows that change between the read and the conditional write. --clear
removes an acknowledgement under the same rules.
The reconciler skips acknowledged blocked rows and counts them as
acknowledged; --include-acknowledged retries them after a fix release.
Other acknowledged rows, such as a stale one, are still processed so they
are saved when they finish, but they are not attention. The attention
alert's instructions now describe both responses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
upload_bytes_immutable gave every attempt a new UUID-suffixed object, so
each retry, each racing writer, and each verification of the same run data
left another copy behind. Storage grew with attempts rather than with
distinct content, and nothing could ever tell the copies apart from
referenced objects.
FileService gains put_if_absent, exists and delete_file:
- GCS uploads with ifGenerationMatch=0, maps a 412 to "exists" beside the
404 mapping, and compares the md5Hash the upload returns;
- local storage writes a temporary file and links it into place, so the
name appears atomically and an existing one is never replaced;
- MinIO/S3 puts with If-None-Match: * (checking a single-part ETag), and
falls back to check-then-put on servers without conditional writes.
get_modified_date now maps a missing object to FileNotFoundError.
publish_content_addressed creates the object, or reuses an existing one
after checking that it decodes to the same canonical SHA-256. An existing
object that does not match is never overwritten: the content is written to
a .repair-<uuid> name and verification_object_integrity_conflict is logged,
which the attention alert now matches.
Every former caller uses deterministic names from object_names.py:
statistics artifacts under verification-activity/{workflow_id}/{digest},
snapshot data and manifests by digest, externalized reports under
verification-reports/{report_id}/{digest}, and restores under the name the
report would have had, so a repeated restore reuses one object.
upload_bytes_immutable is removed.
The tests pin I-6 for publication: ten persist retries create no new
objects, two verifications of the same data share one data object, a
corrupt object takes the repair path, and FakeGcsStorage now honours
ifGenerationMatch. Reverting to UUID names fails eight of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every verification left its statistics artifact and its snapshot index pointer in storage forever, although neither is referenced once the report is stored, and one very large verification could retain unbounded snapshot bytes. After store.save and record_report_persisted succeed, the persist activity deletes, best-effort, the workflow's statistics artifact and snapshot index pointer. A failed delete logs verification_transient_cleanup_failed and is not fatal; the lifecycle template below is the backstop. Because the pointer is gone afterwards, provenance now records snapshot_manifest_reference, which keeps the manifest referenced. Deleting those objects makes a retried persist activity dangerous: a worker that dies after the cleanup would retry, fail to hydrate the deleted statistics, and block a report that is already stored. An attempt that finds the report already stored for the same run and owner therefore only acknowledges it. A test pins this; removing the check fails it. verification_snapshot_max_bytes (64 MiB uncompressed) bounds one verification's snapshots. The budget is checked before any upload. Over it nothing is captured, verification_snapshot_capture_skipped is logged, the pointer records the skip, provenance carries input_snapshots_skipped, and the reconstruct offer is refused as work_limit instead of missing_numerical_inputs. ops/storage/verification-lifecycle.json deletes objects older than 30 days under verification-activity/ and verification-snapshots/index/ only. It is not applied automatically. Its README notes that applying it replaces the bucket's lifecycle configuration, and that without storage.objects.delete only this rule cleans up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deterministic names and transient cleanup stop new garbage, but nothing
removed what had already accumulated: the UUID-suffixed objects of earlier
attempts, manifests whose pointer is gone, and losing report attempts. No
lifecycle rule can collect those prefixes safely, because snapshot data
and reports are long-lived and shared.
scripts/gc_verification_objects.py collects by reachability, dry-run by
default. The reference set is every object a stored report names (its
payload, its snapshots and, now, its manifest) plus everything a live
snapshot index pointer reaches, which covers verifications still
persisting. Candidates are unreferenced objects under
verification-snapshots/{data,manifests}/, verification-reports/ and the
legacy verification-restored/ prefix that are older than --grace (7 days).
--apply deletes them oldest first, capped by --max-deletes; it rebuilds
the reference set just before deleting, so an object reused during the
listing is kept, and it refuses to delete while a live pointer cannot be
resolved. Each run writes a 0600 report, never replacing an earlier one,
and prints per-prefix totals of objects, bytes, unreferenced bytes and
candidates as the storage baseline.
FileService gains iter_objects, which pages through a prefix; on GCS it
follows list_objects' pageToken instead of the unpaged get_listing.
Tests pin that a referenced object is never a candidate however old, that
apply deletes exactly the stale unreferenced objects and leaves the stored
report and its retained inputs whole, and that the cap, the refusal and the
late-reference rescue hold. Removing the report references, the live
pointers or the grace period from the script each fails them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
accept_reconstruction downloaded, decompressed and checksummed up to 200 snapshot objects before it charged the shared workflow budget. A caller already over the limit could still make the API read and parse every retained input of a public verification, on every request. Admission now runs cheapest first: 1. the idempotency peek; 2. the provenance read and the in-memory eligibility and input checks; 3. the shared-budget charge; 4. an existence check of each distinct snapshot reference, 8 at a time within 10s, with metadata calls only; 5. the reservation and the start. _reserve_workflow_id no longer charges; _charge_new_reservation does, only for a reservation that does not exist yet, so an idempotent replay is still free. A missing snapshot is still a typed 409 that starts nothing, but the charge stands because storage was read. A storage error or a blown budget is a 503. Re-verification is charged before its archive lookup for the same reason. _verify_snapshot_bytes is gone; load_verified_snapshots runs only in the comparison activity. Contract change: a checksum mismatch is no longer a 409. The comparison activity raises a typed, non-retryable error, and the derived report fails with failure_code "checksum_mismatch" (or "missing_numerical_inputs") and an exact workflow_error. VerifyWorkflowOutput gains failure_code, and the OpenAPI spec is regenerated. Tests spy on every FileService call: a throttled caller gets 429 with none, an eligible request makes only existence checks, and a missing snapshot is a charged 409. On a real worker, a tampered snapshot gives a derived FAILED report with checksum_mismatch. Moving the charge after the existence check fails the spy test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every GET /verify/{id} built recovery offers: a provenance read, an archive
lookup and a storage metadata call, plus a second ledger read, on every
progress poll of a running verification as well as on every finished
report. The UI then showed recovery buttons on healthy and running reports.
A report response now carries no recovery:
- A verification that is still running gets progress only. A poll makes
no ledger, provenance, archive or storage call beyond what the progress
itself needs.
- A finished report is annotated with its derivation fields and derived
links, using the ledger row the request already read; it is not read
twice.
The new GET /verify/{id}/recovery computes the offers when a caller asks
for them. It has the same visibility as GET: a hidden verification is a
plain 404, and the response is private, no-store. build_recovery takes the
provenance its caller already has instead of reading it again. The
unavailable-original 404 keeps its recovery offers, since that is where they
are offered unprompted.
The optional ETag is not added. A report's content_sha256 does not cover
the derived links annotated onto the response, so a 304 could serve a
stale list.
Tests cover: a 200 has no recovery; the /recovery access matrix (public,
owner, admin, other, anonymous, unknown); a running poll makes no
provenance, OMEX, storage or derived-link call (spies); and a finished
report reads its ledger row once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A public verification's permalink listed its derived verifications oldest
first, the first 100 only. Anyone could start them without limit, so a
flood of requests could push the owner's and admins' links off the page,
and nothing showed who had started a link or whether it used arbitrary
tolerances.
Every derived row now records initiator_relation (owner, admin, other or
anonymous) at reservation time; admin is decided then, since roles change.
It also records custom_settings: whether a re-verification's tolerances or
observables differ from the documented defaults. Links expose the relation,
a per-caller is_yours and custom_settings, never an identity. The derived
report itself carries custom_settings too.
list_derived sorts newest first, takes initiator filters, and returns
{items, total, truncated}, served by a new (source_verification_id,
created DESC) index. The permalink lists:
- every owner- and admin-started link, capped at 10;
- the caller's own, capped at 10;
- the 5 newest started by anyone else.
derived_verifications_total gives the overall count, so the UI can say
"N more". The unavailable-original 404 lists them the same way.
One caller may start at most 10 derived verifications of one source per
day, or 3 anonymously. The count comes from BiosimVerifyIdempotency via a
(caller_key, source_verification_id, created) index and is checked before
the shared workflow budget, so a capped request costs nothing. Exceeding
it is a 429 with code derived_limit. An idempotent replay is neither
counted nor charged.
The optional GET /verify/{id}/derived?cursor= is not added.
Tests: 105 links from others never displace owner, admin or the caller's
own; owner and admin links are capped at 10; both caps hold, per source and
for a day only; the capped request is never charged; custom settings are
flagged; and no initiator identity appears in any response. Ungrouping the
listing, dropping the cap or checking it after the charge, or exposing the
initiator each fails them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verification panel showed "Reconstruct comparison" and "Start
re-verification" whenever an offer was eligible, including on healthy and
running reports. The re-verify form only displayed its defaults, so a
caller could not choose simulators or tolerances.
Where recovery actions appear now:
- An unavailable original shows them automatically, from the 404.
- A finished COMPLETED or FAILED report shows a collapsed "Recompute…".
Opening it calls GET /verify/{id}/recovery once, lazily.
- Nothing shows while a verification is running or saving.
recoveryMode, beside verificationIsDone, decides this in one place.
The re-verify form is editable:
- a simulator multi-select seeded with the defaults and the suggested
versions, which accepts a typed name or name:version;
- relative and absolute tolerances;
- include outputs;
- the description.
Each field shows its documented default. Client-side bounds mirror
ReverifyRequest: 1 to 10 distinct simulators, and finite tolerances of at
least 0. ReverifyRequest now enforces that lower bound too: a negative
tolerance would fail every comparison, and is a 422. An invalid draft is
never sent, and the idempotency key stays unique per submitted body.
Derived verifications are grouped as owner or admin, yours, and others.
The panel says how many more are not listed, and shows a "Custom
tolerances" badge on links and on a derived report whose settings differ
from the defaults. The 200 response no longer carries recovery, so the
panel stops reading it.
frontend/app/models/verification.ts gains the new fields (failure_code,
derived_verifications_total, custom_settings, the link fields and
ReverifyRequest), and the OpenAPI spec is regenerated.
Tests:
- recovery visibility for running, saving, stored, failed and unavailable
reports, in the real component and in recoveryMode;
- the lazy /recovery load happens once;
- edited settings reach the request body, and invalid ones are not sent;
- client-side bounds;
- derived grouping with "N more";
- backend: a negative tolerance is a 422, and a derived report carries
custom_settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report_backup.py reached into the report store's and the ledger's private
collections, object storage and settings to export and restore. Any change
to how either class stores data could silently break backups, which nothing
else would notice until a restore was needed.
The report store gains iter_envelopes, read_artifact, check_envelope and
restore_envelope; the ledger gains iter_rows and restore_row. Backup and
restore now go through those alone. restore_envelope keeps the existing
rules: it never replaces a report, an identical one is left alone, a
different one is an integrity conflict, an externalized payload is
published under its deterministic name, and the restored report is read
back before it returns.
A grep-based test fails if any production module touches another object's
_collection, _file_service, _idempotency, _maintenance or _settings, and if
report_backup.py issues a collection call of its own.
derivation_kind is now DerivationKind | None on the report, its provenance
and the ledger row, and code compares it with enum members instead of
"reconstruction". The enum serializes to the same string, so stored rows
and content hashes are unchanged and rows written as plain strings still
validate; a test pins both. models.py uses `import re` instead of
__import__("re"). The OpenAPI spec is regenerated for the typed field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any exception from the saved-report lookup made GET /verify a 503, so a Mongo blip stopped every progress poll, even for verifications whose state lives entirely in Temporal. The original plan's lookup contract (§5) promised a Temporal fallback for exactly that case. When the ledger or report-store lookup raises a dependency error (anything but ReportCorrupt or ReportUnavailable), GET now continues to Temporal and answers only what Temporal alone can answer truthfully: - a running verification returns its progress query, authorized from the workflow's own owner_sub (401/403, as before reports were saved), with Cache-Control: private, no-store; - a completed result that was blocked or still pending is returned marked pending, so the client keeps polling; - a stored (slim) result, a failed execution, and a workflow Temporal does not know are a 503 with Retry-After: 5. A stored report can only be read from the report store, and the ledger is what decides "no longer retained", so the answer is never a 404 during an outage. Nothing is written while the ledger is down: no catch-up publish, no terminal-status record, no annotation. A corrupt report is still a plain 503, not an outage. The NOT_FOUND ledger fallback's own 503 also sends Retry-After now. The trade-off: stored reports depend on Mongo, and running progress does not. The outage matrix covers: - running progress; - blocked and pending completed results; - a stored result; - the four failed statuses; - NOT_FOUND at describe and at query; - private access for anonymous, other, owner and admin callers; - a report-store-only outage; - a corrupt report. Each case also asserts that nothing was written. Reverting to a plain 503, letting NOT_FOUND 404, serving a stored result from Temporal, recovering a failed execution, or skipping authorization each fails a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runbook, the architecture notes and the backend guide still described
the branch as it was before the review: a 20-attempt persistence retry,
UUID-suffixed objects with no cleanup, a reconciler that exits nonzero on
any row problem, recovery offers on every poll, a checksum mismatch as a
409, and statistics that "never travel through Temporal payloads".
docs/verification-report-retention.md now covers uncapped persistence and
BLOCKED (including that a blocked result must be stored before its
history expires), the corrected payload claim, the data converter change,
the Mongo-outage answers, the compose init service and developer
migration, reconcile exit semantics, events and the acknowledgement
procedure, and the object layout, lifecycle rule and GC procedure.
docs/workflows-architecture.md gains a persistence and history-recovery
section and rewrites linked recovery: the admission order, the checksum
contract, /recovery on demand and the derived-link policy.
backend/CLAUDE.md lists /verify/{id}/recovery and the changed GET and
recovery rows, the new modules, the operations scripts, and the
FileService additions. The root layout lists ops/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the Verification Report Retention plan, "Durable Reports and Linked Reconstruction": Phase A (durable original reports) in
c9e9daaand Phase B (linked recovery) ine26d039, plus the fixes from the second-pass Phase B audit in0407ef3.def76afmakes the reconcile CronJob opt-in and gives it its own scratch volume (see "Deployment safety" below). The fixes from the review of this PR follow inf6078ac…76b155c(see "Review remediation" below). Start with the reviewer guide.Reviewer guide
The PR stays whole (the owner decided not to split it), so it is laid out to be reviewed in parts. Phase A (durable reports) can be signed off before Phase B (linked recovery).
Reading order. Each step names the invariants it must uphold:
biosim_verify/report_codec.py,report_storage.py,object_names.py;common/storage/file_service*.py,gcs_aio.py,content_addressed.pybiosim_verify/activities.py(persist_verification_report_activity,prepare_report,externalize_statistics),workflow_persistence.py,history.py,omex_verify_workflow.py,runs_verify_workflow.pyapi/main.py:get_verify_output,_recover_verification_from_history,_publish_recovered_output,_ledger_fallback_404,_outage_viewbiosim_verify/report_recovery.py,report_backup.py,database.py;scripts/reconcile_verification_reports.py,ack_verification_persistence.py,gc_verification_objects.py,backfill_verification_reports.py,backup_verification_reports.pycompose.yaml,scripts/smoke-infrastructure.sh,.github/workflows/smoke.yaml,kustomize/components/verification-reconcile/,ops/biosim_verify/linked_recovery.py,reconstruct_workflow.py,snapshots.py,comparison.py; the recovery routes inapi/main.pycomponents/verification/VerificationStatus.vue,utils/verification-status.ts,utils/verification-recovery.ts,models/verification.ts,pages/utilities/verify-model.vueThe docs follow the code:
docs/verification-report-retention.md(runbook),docs/workflows-architecture.md,backend/CLAUDE.md.Phase A and Phase B files. Phase A is reading-order steps 1–5, plus the status/availability part of step 7 (
verification-status.tsand the "Saving report" and "Original report unavailable" states). Phase B is step 6 and the rest of step 7: recovery actions, derived links and catalog badges. Four files hold both:models.py: Phase B adds the derivation, recovery and link models;database.py: Phase B adds the idempotency reservations,list_derivedand the caps;activities.py: Phase B adds snapshot capture andlink_derived;api/main.py: Phase B adds the/recovery,/reconstructand/reverifyroutes and the annotation.Review fix commits, by finding. B = blocker, S = smaller issue, X = found during remediation.
f6078acFix local Temporal volume ownership6f2f7abMake smoke fail when Temporal is unavailablee9e46dbUse persistence-state enums and classify zlib errors as corrupt263b1f9Move history helpers out of the API and make terminal failures deterministichistory.py), X-122e60f0Retry report persistence until it succeeds or is blockeda304e21Keep statistics out of workflow results and queriesec785d4Test payload compatibility with the 0.10.3 data converter7f2c032Separate reconciler job health from data attention363609dAdd persistence acknowledgementa9af171Add create-only, content-addressed object publication72f4ea3Remove transient verification objects and bound snapshot captureed919e2Add verification object garbage collection4528f82Admit reconstruction before any storage work510d376Serve recovery offers on demand8c9cde3Bound and rank derived verification links509f6eeShow recovery actions only where they applyc740715Give backup public persistence APIs and type derivation kinds0a5981cKeep progress polling available during a Mongo outage76b155cDocument the verification retention review remediationWhere the tests are
test_persistence_failures(25 transient failures then stored, permanent →blocked, terminated while persisting, unreadable result never replaced)test_persistence_failures(terminated while computing: identical digests; losing writer gets 200; stored-but-missing is 503),test_historyprepare_reportskipped; fresh provenance on recoverytest_workflow_payloads,test_generate_statisticstest_recovery_admission,test_recovery_offers,test_derived_linkstest_reconcile_job,test_persistence_ack,test_verification_reconcile_componenttest_object_storage,test_transient_objects,test_object_gc,test_file_service_create_only,test_gcs_missing_objectstest_compose_drift,test_smoke_infrastructure, the smoke workflowdepends_onremoved; the olddef76afcompose (see "Smoke failure evidence")test_verification_outage(16 cases, each asserting no writes)test_converter_compat,test_layeringThe full mutation record, per phase, is under "Tests". The six mutations the remediation plan's validation matrix requires were re-run on the final SHA (see "Final validation").
Disclosures
pydantic_data_converter; 0.10.3 used the SDK's defaultDataConverter.test_converter_compat.pyproves it compatible with every payload type 0.10.3 could write, and reads and replays a history written by the old converter, so backfill can read retained 0.10.3 histories. No decoding shim is needed.GET /verify/{id}still serves running progress and never-stored results from Temporal. A stored report, a failed execution and an unknown id are a503withRetry-After: 5, never a404. Stored reports depend on Mongo, and running progress does not.Problem
Verification reports existed only in Temporal workflow history. The ledger (
BiosimCompare) stored a verification's identity, owner, archive hash and terminal status, but not the report or the request that produced it. Once Temporal removed the history,GET /verify/{id}could only answer404 "Verification <id> exists but its results are no longer retained."defaulthas 24h retention, and history and visibility archival are disabled (checked directly on 2026-10-29).compose.yamlran Temporal in-memory, so restarting the container lost every report.PENDINGfor a missing report and then stopped polling, leaving a futile Retry button.Recovery must never overwrite a historical answer. There is no same-ID rerun, no report reset, and no silent recomputation on GET.
Phase A: durable original reports (
c9e9daa)Immutable report storage (
BiosimVerifyReports)report_codec.py: theverification-json-gzip-v2codec keeps NaN and ±Inf scores and literal dotted dataset keys. The SHA-256 covers the canonical uncompressed bytes.report_storage.py: statistics above 512 KiB are externalized to object storage before they cross Temporal. After persistence, workflow results and queries carry no statistics. Objects get deterministic, content-addressed names and are only ever created, never overwritten, so a retry reuses the object it already wrote; the report is then published through an insert-once manifest. An identical retry is a no-op; conflicting content, owner or run IDs cannot replace a published report.Settings: 12 MiB inline report, 64 MiB decompressed ceiling (corrupt or oversized payloads are rejected), and 512 KiB for an activity's inline payload (measured on the uncompressed JSON).Publication before completion
persist_verification_report_activitystores the report before the ledger pointer is updated or completion is acknowledged.OmexVerifyWorkflowandRunsVerifyWorkflowgate the new commands behindworkflow.patched("durable-verification-report-v1"). Runs statistics also sit behinddurable-verification-statistics-v1(generate_statistics_activity_v2). In-flight and pre-change histories therefore replay, and replay is tested.workflow_persistence.py).blocked. Permanent failures are aValueError, an integrity conflict, a corrupt artifact or a missing artifact.persistence_state=blocked. A successful verification is never reported as FAILED.history.terminal_failure_output), with honest missing-result provenance.Lookup:
GET /verify/{id}reads the saved report firstWorkflowHandle.result(), with no worker query, then authorized and persisted as a catch-up write. Ifresult()can't be read, GET returns503and stores nothing.persistence_state=pending.200). If that report can't be re-read, GET returns503and logsverification_persistence_attention. It never returns a 500.storedwhose report is missing is a503with an attention log. The slim Temporal result is never served as the report.503, not a 404. A missing report is reported separately, and hidden rows still get the generic 404.private, no-store.503withRetry-After: 5, never a 404.detailstring for older clients.TEMPORAL_NAMESPACEsetting defaults todefault.Operations
scripts.reconcile_verification_reportsrepairs missed persistence and terminal failures without needing a user to call GET. Its exit status reflects the job's health only; rows that need an operator raise an alert, andscripts.ack_verification_persistenceacknowledges them without touching a report (see "Review remediation").verification-report-reconcileCronJob runs it every 5 minutes:Forbidconcurrency, 240s deadline, 150s scan budget, 45s per-row budget. It ships as the opt-in kustomize componentkustomize/components/verification-reconcile, which no overlay enables yet.BiosimVerifyMaintenancefence concurrent jobs, and a job killed at its deadline resumes where it stopped.scripts.backfill_verification_reportsis resumable and dry-run by default, with 0600 checkpoints, bounded pages, and--since/--until. It never starts workflows or simulations, and it never invents ownership. Rows with unknown owners are reported for investigation.scripts.backup_verification_reports export|restoreexports ledger/request BSON, report envelopes, referenced objects, and a checksum manifest. Restore checks every checksum and report identity before writing (--applyto write). Backup and restore go through the report store's and the ledger's public methods only.ops/monitoring/holds alert templates (not installed):verification_persistence_attention,verification_persistence_failureandverification_object_integrity_conflict;verification_reconcile_summaryheartbeat;ops/storage/verification-lifecycle.jsonis a GCS lifecycle rule for the two transient prefixes (not applied automatically), andscripts.gc_verification_objectscollects unreferenced long-lived objects, dry-run first.docs/verification-report-retention.md: drain, backfill, Temporal recreation, backup and restore drill, and alert response.Local Temporal persistence:
compose.yamlnow runsstart-dev --db-filename /data/temporal.dbon a newtemporal_datavolume. A one-shottemporal-data-initservice hands the volume to the image's uid/gid 1000 (f6078ac); without it the server could not open its SQLite file. Retention still applies; this only makes local history survive a restart or recreation.Frontend:
verification-status.tsandVerificationStatus.vueseparate workflow status from report availability, showing "Saving report", "Original report unavailable", or a service error. Polling stops for lookups that can't recover. The component guards against late responses after the workflow ID changes and cancels on unmount.Phase B: linked recovery (
e26d039,0407ef3)Both routes require an
Idempotency-Keyheader and return202with a new ID. The source's ledger row, report bytes, content hash, status and permalink stay unchanged.POST /verify/{id}/reconstructReconstructComparisonWorkflow(new ID prefixreconstruction-)POST /verify/{id}/reverifyOmexVerifyWorkflow(new ID prefixreverification-)Comparison-only reconstruction
snapshots.py). They are content-addressed by SHA-256 and published through a per-workflow pointer. A failed capture logsverification_snapshot_capture_failedand leaves the original report intact; it only makes that report ineligible for reconstruction.ReconstructComparisonWorkflowrunscompare_retained_snapshots_activityand then the shared persist activity. Its module never referencessubmit_biosim_run_activity,get_hdf5_dataorget_existing_biosim_simulation_run_activity, so it works while biosimulations.org is down. A missing or mismatched snapshot raises a non-retryableApplicationError.409s that reserve no ID and start nothing:missing_provenance,missing_numerical_inputs,input_mismatch(the client must echo the storedrun_ids,snapshot_sha256andcompare_settings), andwork_limit. Admission only checks that the retained inputs exist. A checksum mismatch is caught by the comparison, and the derived verification then fails withfailure_code: checksum_mismatch.computed_at, and its source. It is labelled a new calculation, not a restoration.Archive re-verification: this needs a stored, readable archive. The offer shows the documented defaults (simulators and tolerances) and says plainly that it can start new simulations. Rows that only have runs, with no stored archive, are refused.
Shared submission contract
BiosimVerifyIdempotency.409 idempotency_conflict.503, and a retry reuses the reserved ID.429with codederived_limit, and it is not charged. Retries cost nothing. Existing input bounds apply, for example more than 10 simulators, or a negative tolerance, is422.owner_sub, so a private source never produces a public result. They also storesource_verification_idandderivation_kind, and record the caller asinitiator_subfor audit and quota.initiator_subis omitted from listings and public output. Hidden sources return a plain 404.derived_verifications_totalgives the full count.initiator_relation, plusis_yours, never as an identity, and flagscustom_settings.recoveryandderived_verificationsare sibling fields ofdetail, sent withCache-Control: private, no-store. A caller who can't see the row gets neither field.GET /verify/{id}/recovery, never from the report response.Frontend
crypto.getRandomValuesand are unique per request body.verify-model.vue) shows a "New calculation" or "Re-verification" badge with the source ID.Fixes from the second-pass audit (
0407ef3)GCS missing objects. gcloud-aio-storage reports a missing object as an HTTP 404
ClientResponseError, never asFileNotFoundError.gcs_aio.pynow maps that status, and only that status, onto theFileServicecontract:Nonefromget_file_contents, andFileNotFoundErrorfromget_modified_date. Before this fix, on GCS:Other HTTP errors are still raised.
The unavailable-original 404 now carries
derived_verifications, and the UI renders them.The client's idempotency key is now random and unique per request body.
The OpenAPI spec is regenerated, and
frontend/app/models/verification.tsis updated with the new additive optional fields. Readers tolerate old rows that have no inputs, provenance, or stored report.Review remediation (
f6078ac…76b155c)A review of this PR found these problems:
Phases 1–5 of the remediation fix them, and Phase 6 documents them.
Phase 1: CI honest and green
f6078ac: fixes local Temporal volume ownership (see "Local Temporal persistence" above).test_compose_drift.pypins the image, the init step and the health checks without Docker.6f2f7ab: smoke now fails when Temporal is unavailable.SERVING.checks.temporal == truefrom the API.compose.yaml, it exits 1. Evidence is indocs/pr-163-phase-1-validation.md.e9e46db: persistence code usesReportPersistenceStatemembers, so the serializer no longer warns. A corrupt DEFLATE stream (zlib.error) is now classifiedReportCorrupt.Phase 2: a computed result is never degraded, writers converge, payloads are bounded
biosim_verify/history.py, so the reconcile CronJob no longer imports the FastAPI app.activities.prepare_report(). Recovered reports keep their snapshots, run order and re-verification link, and their recorded provenance is markedrecovered_from_history.blocked, never FAILED (see "Publication before completion").ReportArtifactMissing) is permanent. Unconfigured or unreachable storage stays retryable.storedledger row is never marked unavailable afterwards.None. A history written by the old default converter is also read and replayed under the new one. No decoding shim is needed.Reading the Phase 2 commits one at a time: all Phase 2 production code is in
263b1f9.22e60f0,a304e21andec785d4add only tests and a fixture, although the messages of22e60f0anda304e21describe the persistence and inline-statistics code that is actually in263b1f9.263b1f9and22e60f0do not pass on their own, because a fixture and two test updates land in later commits.a304e21.Phase 3: the reconcile job reports its own health; storage grows with distinct content, not retries. Each Phase 3 commit passes the full backend suite on its own.
7f2c032Job health, not row health.verification_reconcile_summary, the heartbeat. It addsverification_persistence_attentiononly when a row needs an operator.skipped="lease_busy".verification_reconcile_job_failed. That event carries the reason and the exception type, never the message.restartPolicy: NeverandbackoffLimit: 0, so the next 5-minute run is the retry.363609dAcknowledgement.persistence_ack = {state, error, reason, acknowledged_by, acknowledged_at}on the ledger row. It applies only while the row's state and error are unchanged, so a different failure alerts again.scripts.ack_verification_persistenceis dry-run by default. It refuses stored rows, rows not in--expect-state, and rows that change before the write.blockedrows;--include-acknowledgedretries them after a fix release.a9af171Create-only, content-addressed objects.FileServicegainsput_if_absent,existsanddelete_file:ifGenerationMatch=0and treats a 412 as "exists";If-None-Match: *, with a check-then-put fallback.publish_content_addressedcreates or reuses an object. An existing object at that name whose content doesn't match is never overwritten: the bytes go to a.repair-<uuid>copy, and the run logsverification_object_integrity_conflict.upload_bytes_immutableis gone. Ten persist retries create no new objects, and two verifications of the same run data share one object.72f4ea3Transient cleanup and a snapshot budget.work_limit.ed919e2Garbage collection.scripts.gc_verification_objectscollects by reachability.--applydeletes at most--max-deletes. It rebuilds the reference set first, and it refuses to delete while a live pointer can't be resolved.Phase 4: recovery is admitted before it costs anything, offered only where it applies, and bounded. Each Phase 4 commit passes the full backend suite on its own.
4528f82Admission before storage work.failure_code: checksum_mismatchinstead of a 409.510d376Offers on demand.recovery. The newGET /verify/{id}/recoveryserves it, with GET's visibility rules andprivate, no-store.8c9cde3Bounded, ranked derived links.initiator_relationandcustom_settingsare recorded at reservation.509f6eeRecovery UI.ReverifyRequest.Phase 5: hygiene and availability. Each Phase 5 commit passes the full backend suite on its own.
c740715Public persistence APIs; typed derivation kinds.iter_envelopes,read_artifact,check_envelopeandrestore_envelopeon the report store, anditer_rowsandrestore_rowon the ledger.derivation_kindis aDerivationKindenum everywhere. Old string rows and content hashes are unchanged.0a5981cProgress during a Mongo outage, as described in "Lookup" above. A 16-case matrix asserts each answer and that nothing is written.Phase 6: documentation.
76b155cchanges docs only.docs/verification-report-retention.mdnow covers uncapped persistence andblocked, the corrected payload claim, the converter change, the outage answers, the compose init service, reconcile exit semantics and events, the acknowledgement procedure, and the object layout, lifecycle rule and GC procedure.blockedresult is in a completed workflow's history, which retention removes after 24h in production. The fix has to be released and the row retried before then.docs/workflows-architecture.mdgains a persistence and history-recovery section and rewrites linked recovery: the admission order, the checksum contract,/recoveryand the derived-link policy.backend/CLAUDE.mdgains/verify/{id}/recovery, the changed GET and recovery rows, the new modules and scripts, and theFileServiceadditions.Tests
New suites:
test_report_codec,test_report_storage,test_report_recovery,test_retention_operations.test_retention_crash: real SIGKILL of a worker before and after each of upload, manifest commit, ledger update and activity ack.test_retention_integration: old/new replay, backfill pagination, Temporal restart and recreation, reconnect after an outage, history deletion, and an independent restore.test_linked_recovery: 19 tests, including a real worker and real Temporal with upstream patched to fail, plus idempotency convergence and lost replies.tests/common/test_gcs_missing_objects.py.verification-statusandverification-component. The component tests mount the real<script setup>in avmsandbox.test_compose_drift;test_smoke_infrastructure;test_history(9 tests);test_persistence_failures(21): 25 transient failures then stored, permanent failure toblocked, terminated while persisting or computing (including concurrent writers), the losing writer, stored-but-missing, and replay of the capped-policy history;test_workflow_payloads(4);test_converter_compat(18);test_reconcile_job(18) andtest_verification_reconcile_component(2): exit codes and events, staleness thresholds, and the CronJob and alert filters;test_persistence_ack(13);test_file_service_create_only(9), plus 9 new cases intest_gcs_missing_objects: the 412 mapping, checksums, idempotent deletes, andpageTokenpaging;test_object_storage(7): retries, shared data, the repair path, and repeated restores;test_transient_objects(6);test_object_gc(9);test_recovery_admission(10): FileService spies on throttled and eligible requests, a charged 409, and a checksum mismatch on a real worker;test_recovery_offers(10): norecoveryon a 200, the/recoveryaccess matrix, and no work on a running poll;test_derived_links(15): ranking, caps, custom settings, and no initiator identity;verification-recovery(4) and 4 new component tests: the visibility matrix, edited settings in the request body, and derived grouping.test_layering(5): the private-state grep, backup through public methods only, string rows still valid, an unchanged serialized form, and redaction;test_verification_outage(16): running, blocked and pending results, a stored result, the four failed statuses, NOT_FOUND at describe and at query, the owner matrix, a report-store-only outage, and a corrupt report.Final validation (
76b155c)The final SHA is the current head;
76b155cchanges docs only. All five checks are green there:uv run python -m pytest): 1,801 passed, 0 failed, 17 skipped. The skipped tests need external credentials (GCS, SLURM, SSH). The log has zeroPydanticSerializationUnexpectedValue.lint0 errors (the 2 existingImageCompare.vuewarnings),typecheck,test:verification41,test:auth5,test:legacy-proxies5.temporal SERVING,temporal persistence probe survived container recreationandbackend reports checks.temporal == true.Checked locally at
76b155c:ruff check .andmypy biosim_server tests(240 files) are clean.scripts/generate_openapi.pyshows no drift.compose.yamlin a disposable Compose project, changing only container names and ports. It exits 0. Mongo and Temporal endhealthy, the init container exits 0, the server runs as uid/gid 1000, and the probe workflow survives--force-recreate.components: [../../components/verification-reconcile]added to a copy ofbiosim-gke, the render gains only theverification-report-reconcileCronJob:backoffLimit: 0,restartPolicy: Never, 240s deadline,Forbid, the overlay's namespace, itsplatform-apitag, the hashedapi-config, and anemptyDirscratch volume.Mutation checks re-run here. These are the six that the remediation plan's validation matrix requires. Each was applied in its own worktree, failed its tests, and was restored. Without mutations, the 73 tests involved all pass.
maximum_attempts=20test_transient_failures_are_retried_until_the_report_is_storedtest_terminated_while_persisting_both_writers_store_the_computed_output(inline and externalized)test_terminated_while_computing_writers_store_one_identical_failuretest_a_throttled_reconstruction_does_no_storage_work,test_a_missing_snapshot_is_refused_and_stays_chargedtest_reconcile_jobandtest_persistence_acktest_object_storage(reuse, sharing, retries, repair, restore) andtest_oversized_statistics_crosses_activity_boundary_as_referenceCI runs: backend-ci, frontend-ci, smoke.
Each Phase 3, 4 and 5 commit passed the full backend suite on its own. Phase 2's
263b1f9and22e60f0do not (see "Reading the Phase 2 commits").After #162 merges, this PR is retargeted to
main, andorigin/mainis merged in (or the branch is rebased if #162 is squash-merged). This validation is then repeated on the resulting SHA.Mutation record
Each mutation was applied, its tests were run and failed, and the code was restored.
prepare_reportskipped, fresh provenance on recovery, the persist result keeping statistics, the ledger read swallowed, and the X-1 guard removed.recoveryback on a 200, an ungrouped derived listing, no daily cap or a cap checked after the charge, and a link that exposes its initiator.0407ef3: making the same-start confirmation always refuse; dropping derived links from the backend 404 or from the frontend's 404 handling.Smoke failure evidence (remediation step 1.7)
The smoke entry point (
scripts/smoke-infrastructure.sh infra, the one CI runs) was run on macOS (Docker Desktop 29.8.2) against the olddef76afcompose.yamlon a fresh named volume, in a disposable Compose project. It exits 1:SQLite cannot open its file because the fresh volume is root-owned and the server runs as uid 1000. With the fixed compose, the same entry point exits 0, and CI's smoke log shows
temporal SERVING,temporal persistence probe survived container recreationandbackend reports checks.temporal == true. Removing the initdepends_onfailstest_compose_drift.py. Full record:docs/pr-163-phase-1-validation.md.Deployment safety (
def76af)Merging this PR deploys nothing to any cluster. All four overlays (
biosim-gke,biosim-rke,biosim-local,biosim-rke-frontend-dev) render byte-identical to the PR base withkubectl kustomize.kustomize/base, which every overlay includes. Flux would have created it inbiosim-gkewithplatform-api:backend-0.10.3, which doesn't containscripts.reconcile_verification_reports, so it would have failed every 5 minutes. It is now the componentkustomize/components/verification-reconcile. The deploy PR that moves an overlay to a backend release containing the module addscomponents: [../../components/verification-reconcile]to it. Rendered with the component enabled, the CronJob picks up the overlay's namespace, itsplatform-apitag, and the hashedapi-configname, and nothing else in the overlay changes./app/scratchis now anemptyDir(1Gi limit). It previously mountedworker-cache-pvc, which isReadWriteOnceonbiosim-gkeand held by the worker, so a job scheduled on another node could never attach it. The path only holds one run's temporary downloads.The runbook,
backend/CLAUDE.md(deploy step 5) and the rootCLAUDE.mdlayout describe the component.newTagbump, as described above.backend/CLAUDE.md, includingflux suspendbefore the overlay merge. Run backfill while the 24h histories are still retained, and again after the drain. The patches keep old histories replayable; the drain is still required.pydantic_data_converter. 0.10.3 used the SDK's defaultDataConverter. Compatibility with 0.10.3 payloads and histories is tested (test_converter_compat.py), so backfill can read retained 0.10.3 histories. The worker drain in step 2 is still required for the workflow-code changes.docker compose uprecreatesplatform-temporalwith the new volume. Existing in-memory history is not migrated. Do not rundocker compose down -v.Known limitations (recorded, not changed)
blockedone), it still builds fresh provenance withoutsource_verification_idandderivation_kind. Reports recovered from history now keep them. Display is unaffected, because the link is read from the ledger row.stored, has to raise an unclassified error: Mongo dropping between the two reads. A failure of the first lookup is the outage path above.render = () => null.Still open after merge (production acceptance)
ops/storage/verification-lifecycle.json. Applying it replaces the bucket's lifecycle rules, so merge it with the existing ones. Confirm the service account hasstorage.objects.delete; without it, only the lifecycle rule removes transient objects.scripts.gc_verification_objects(dry-run) against the production bucket and record its totals as the production storage baseline. The local baseline is all zeros, because the local store holds no verification objects.🤖 Generated with Claude Code