Distinguish native cancellation and preserve bounded recovery receipts

Author: KSemenenko

ai-database-sql.acceptance.md

@@ -63,3 +63,28 @@ Schema-name clarification before source join: Q1 metadata names `revision`, `*`
 Pre-delivery public enum spelling is WholeNumber for signed Int64 and FixedPoint
 for exact decimal, with unchanged ordinal/range/scale contracts. The initial
 candidate's CA1720 failures are tracked in the plan and repaired without suppression.
+
+AC-AISQL-013 / AC-ROUTE-010 refines the retained-replica failure contract: an
+OperationCanceledException inside a read actor is caller cancellation only when
+that actor's actual incoming token is cancelled. An active caller token returns
+Cancelled; an inactive/absent caller token returns OwnershipLost with fixed safe
+unavailable detail. Possibly dispatched commands retain UnknownWriteOutcome for
+both cases; typed domain errors retain exact codes/details. RequestGrain,
+DatabaseReadGrain and CommandPartitionGrain must pass their actual incoming token
+to the one reply classifier. No internal retry or accepted-error expansion.
+Automated proof: focused TUnit cases using genuine cancelled/uncancelled CTS and
+actual framework cancellation exceptions, existing domain/privacy diagnostics
+regressions, and unchanged stopped-replica RF3 catch-up through real SDK. Required
+exact-SHA build/formatter/unit/recovery/RF3 joins remain AC009. This corrects a
+concrete source classification defect; the prior RF3 causal phase is unproven.
+
+AC-AISQL-014 / existing AC-RC-003: seeded crash recovery emits the unchanged
+success receipt immediately after real atomic/durable assertions inside the
+original trial try, before unconditional cleanup. Preserve the original linked
+15s token, all20x50 native trials/fault points/assertions, actual four-slot
+occupancy and cleanup/permit exception semantics. Receipt failures receive the
+existing seed/trial/stage diagnostic catch; cleanup failures still fail the test.
+No timeout, accepted-error or fault assertion is relaxed. Automated proof is the
+unchanged genuine seeded CrashHost cases and full three-OS recovery in GitHub,
+including exact previously failing batch7 plus1000 success rows perOS; rare
+external scheduling/cancellation uses existing source/lifetime review exception.

ai-database-sql.plan.md

@@ -27,6 +27,9 @@ no reliable cheaper-tier correctness/routing evidence is available in this run.
 | TASK-AISQL-012A; Accepted ADR-054 integration repair | models worker; Client/Features/BlobStorage/BlobClient.cs and NEW UnitTests/Features/BlobStorage/BlobSdkArgumentTests.cs only; root owns Send access join | 012,006,009 | Tests-first public null guards, source-compatible feature extension methods, same transport/IDs; root review + exact build/RF3 |
 | TASK-AISQL-012B; Accepted ADR-054 integration repair | gates worker; QueryCandidateReader.cs only | 008,009 | Remove depth4 nesting without new allocations/semantic changes; existing25 real-store regression cases + exact build |
 | TASK-AISQL-018; root static scenario review | models worker; RelationalSchemaTests.cs only | 002,004,009 | Ensure schema-change fixture is independently valid (retain indexed name column), then assert exact UnsupportedCapability and unchanged persisted schema; malformed-index validation remains separately tested. No production precedence change or assertion weakening |
+| TASK-AISQL-023; existing AC-RC-003/AC014 contract before code | gates worker; RecoveryTests.cs RunSeededCrashTrialAsync call placement only | 014,009 | Receipt immediately after successful real atomic/durable assertions inside originaltry before mandatorycleanup, same15slinkedtoken/20x50trials/4slots; existingcatch capturesreceiptfailureidentity, no newtimeout/skip/looserassertion; exact3OSrecovery+1000receiptsperOS join |
+| TASK-AISQL-020; Accepted ADR-036/AC013 refinement before code | SQL worker; GrainReplyFactory.cs, RequestGrain.cs, DatabaseReadGrain.cs, CommandPartitionGrain.cs and new GrainReplyCancellationTests.cs only | 013,011,009 | Genuine caller-token cancellation tests first, sole classifier and3 actual token joins; preserve domain errors, command unknown outcomes, current closed-category logging/privacy and unchanged retry assertions; root reviews and exact-SHA complete gates join |
+| TASK-AISQL-019; exact formatter diagnostics | root; SqlRf3DeliveryTests.cs and OrleansRpcFailureTests.cs only | 006,011,009 | Split object initializer members and explicitly qualify the one native messaging exception to remove conflicting import ordering; no semantic/test changes; exact-SHA formatter join |
 | TASK-AISQL-015; exact UnitTests compiler diagnostics | models worker; RelationalLinkageTests.cs, RelationalTestData.cs and RelationalRowTests.cs only | 002–004,009 | Correct actual ordered TUnit collection comparison and cache CompositeFormat without altering exact raw decimal fixtures or assertions; exact-SHA compiler/units join |
 | TASK-AISQL-016; exact UnitTests compiler diagnostics | SQL worker; OrleansRpcFailureTests.cs, SqlOperationBudgetTests.cs and SqlOperationCompilerTests.cs only | 005,007,011,009 | Await native CancelAsync before cancellation assertions and remove imports proven redundant by actual compiler; same errors and test coverage, no suppression |
 | TASK-AISQL-014; exact RF3 compiler diagnostics | gates worker; SqlRf3AdmissionTests.cs only, root owns static scenario member | 004,007,009 | Isolate existing per-node using/await-using ownership in VerifyNodeAsync outside fixture try/finally/loop analysis; start caller deadline after separately bounded fixture readiness; preserve all3 real-node assertions, same-ID control progress and schema/linkage contracts; no suppression/test weakening |
@@ -183,3 +186,19 @@ Candidate3a744d19aa4443a522d303d8d91257f2e20396da / [37070004864](https://github
 Root source review found the schema-migration test removed the indexed name column while retaining its index. ValidateResource correctly rejects that malformed schema before checking migration. TASK-AISQL-018 changes only the intended valid changed-schema fixture and proves it validates independently; migration refusal and unchanged original persisted schema must remain exact. CI3a744d19a is already running, so its observed result remains authoritative and the repaired fixture needs a later exact-SHA run.
 
 TASK-AISQL-017 independent review is complete/clean for all11 original compiler diagnostics. TASK-AISQL-018 is source-complete and root-reviewed: valid nullable Name change with preserved indexes, explicit schema validity, exact migration refusal and byte-for-byte persisted-resource equality. GitHub execution remains required.
+
+Candidatec3eeb2b7a512f31e3489cef44b6127d9d280e836 / [37070513242](https://github.com/managedcode/KeyLoad/actions/runs/37070513242) includes reviewed TASK-AISQL-018 and is queued behind3a744d19a. The3a candidate has passed Linux/macOS solution builds, analyzer-rule tests and IntegrationTests compilation; RF3 runtime/formatter/remaining verification are in progress. These partial successes do not qualify the complete stage.
+
+Candidate3a744d19a / run37070004864 Ubuntu job111047630094 passed full build and real process recovery136/136 with no skips. Formatter failed on SqlRf3DeliveryTests95/96 two same-line initializer members and OrleansRpcFailureTests import order. Units/scalar units were skipped after format failure, so they are not qualified. TASK-AISQL-019 source fixes only those exact diagnostics; full gates must repeat.
+
+Actual3a744d19a RF3 artifact SHA matches. Job111047630080 passed59/60, no skips; all14 new SQL/relational scenarios passed. Existing retained-replica catch-up failed17.976sec with server Cancelled detail, not SDK write timeout. Required accepted retry errors are unchanged. Investigate caller-token vs internal native read cancellation distinction; initial-RPC source correction does not by itself prove the original cause. Receipt: docs/implementation/ai-sql-qualification-37070004864.json.
+
+TASK-AISQL-020 approved before implementation: server-sourced Cancelled in retained-replica test exposes read OCE classification without incoming-token context. SDK write transport mapping is already correct. Per-fixture failure names can overwrite prior tails, so exact causal phase remains unproven; concurrent diagnostic work is preserved. No native-helper expansion, SDK workaround or broader election retry acceptance.
+
+Run37070004864 completed failure: complete three-OS builds/analyzer-rules pass, known formatter failures; Linux process recovery136/136, macOS recovery passes pendingcount, Windows recoveryfails under read-only audit. RF359/60 all14newpass; comparisonsfail under audit. Supersededc3eeb2b7a run37070513242 cancelled because parent formatter/cancellation defects remain and reviewed repairs must be qualified together; cancellation/unfinished jobs never count as passing.
+
+TASK-AISQL-021 read-only audit complete: Windows135/136, failurebatch7/seed1708 at evidence.WriteLineAsync aftersuccessfulatomicassertions and cleanup. The actualdeadline source/trial remainunproven. TASK-AISQL-023 accepts existing Receipt-before-Cleanup contract correction only; no storage/dependency fault is inferred or hidden.
+
+TASK-AISQL-022 completed read-only retained-artifact audit: comparison reports bind exact3a; smoke96cases46measured48unsupported2failed (KeyLoadStreamAppend native adapter defaultReadEventAsync),552measuredsamples succeeded, TimeSeries20attempts+20correctnesspass but both publicAspiretestsWaiting/noCreation/Health. Job2/4passing/no skips. Committedcomparisoncaller/compositionunchangedfrombaseline; concurrentowningworkpreserved. No SQL-specific performance claim or passing comparison gate.
+
+TASK-AISQL-020/023 source artifacts are complete and root-reviewed; independent gates review is clean. Source corrections preserve every domain error, actual actor token, unknown command outcome, native diagnostics privacy, original crash trial bound/receipt fields/cleanup and unchanged RF3 retry expectations. TASK-AISQL-019 exact formatter source repair joined. No complete runtime gate is inferred; new exact-SHA CI and1000 native crash receipts perOS remain required.

docs/ADR/ADR-036-orleans-foundation.md

@@ -542,3 +542,36 @@ the existing evidence-directory/report-copy methods into the new internal
 ComparisonTestEvidenceFiles.cs helper under the same slice, preserving every
 path, filename and call order. These are source-quality preserving joins;
 the bounds/privacy/manual environmental evidence contract above is unchanged.
+
+### Incoming cancellation context refinement
+
+Accepted TASK-AISQL-020 under AC-AISQL-013, REQ/AC-ROUTE-010 and existing
+REQ/AC-ROUTE-008/009. A read actor must distinguish actual incoming-token
+cancellation from an internal native cancellation; the latter returns
+OwnershipLost rather than asserting that the caller cancelled. Commands retain
+UnknownWriteOutcome, and typed domain errors remain authoritative.
+
+```mermaid
+flowchart LR
+    Fault[Actor cancellation exception] --> Intent{Command}
+    Intent -->|yes| Uncertain[UnknownWriteOutcome]
+    Intent -->|no| Token{Incoming token cancelled}
+    Token -->|yes| Caller[Cancelled]
+    Token -->|no| Unavailable[OwnershipLost]
+```
+
+Implementation order: approve acceptance; author genuine cancellation and typed
+domain regression sources; extend the sole GrainReplyFactory classifier with
+incoming token context; pass each actual token from RequestGrain, DatabaseReadGrain
+and CommandPartitionGrain; root reviews then joins exact-SHA complete GitHub gates
+and the unchanged retained-replica test. SQL worker owns those four ClusterRouting
+files plus new UnitTests/Features/ClusterRouting/GrainReplyCancellationTests.cs;
+root owns docs and delivery. No other worker changes these files.
+
+Dependencies: existing native request isolation, transport and read-barrier
+contracts. No persistent schema, wire-envelope, credential, topology, retry or
+journal migration. Rollout deploys the source together; rollback reintroduces
+ambiguous cancellation classification and requires requalification. Keep the existing private closed-category logging contract; the diagnostic classifier
+must receive the same computed code as the returned reply. Internal execution
+phase of run37070004864 failure was not retained reliably, so this refinement is
+source-justified without claiming that its RF3 root cause is established.

docs/Features/ClusterRouting.md

@@ -107,3 +107,13 @@ tracking regardless of the implementing assembly name. Replica Grain Services ru
 before membership is Active and cannot depend on ordinary graph telemetry grains.
 The owning repository's regression, patch release, GitHub publication and intended
 NuGet feed verification are mandatory before changing KeyLoad's package pin.
+
+REQ-ROUTE-010 maps to AC-ROUTE-010 / AC-AISQL-013: read cancellation replies use
+the actual incoming actor token. Only active caller cancellation returns
+Cancelled; inactive-token native cancellation returns OwnershipLost, and commands
+retain UnknownWriteOutcome. Typed domain errors, safe details and existing closed-category
+logging/privacy remain authoritative. ADR-036 TASK-AISQL-020 owns the sole
+classifier and all three actual token joins. Automated evidence: new genuine
+GrainReplyCancellationTests plus unchanged retained-replica SDK RF3 catch-up;
+exact-SHA qualification is pending. This corrects a concrete source defect without
+inferring the prior run's exact internal phase from overwritten fixture logs.

docs/Features/StorageRecovery.md

@@ -219,3 +219,20 @@ alone proves no historical recovery, power-loss guarantee or performance gain.
 ProcessDurable та QuorumProcessDurable описують перевірений declared process failure model історичного коду; `LocalDurable`/`QuorumDurable`, power-loss і endurance не приймаються із process-kill source/test names. Full ACK barrier — [ADR-003](../ADR/ADR-003-durability-ack-barrier.md); committed read views — [ADR-004](../ADR/ADR-004-committed-read-views.md); versioned keyspace — [ADR-005](../ADR/ADR-005-canonical-keyspace-codec.md); backup/log retention — [ADR-008](../ADR/ADR-008-backup-log-retention.md), [BackupRestore](BackupRestore.md).
 
 Current-code journal/compaction/checkpoints і in-progress scoped visitor ремонти не є proof нового delivered SHA. Все source/recovery evidence кваліфікується тільки в GitHub TUnit, actual child-process recovery і RF3 suites. Future maintenance автоматизація, широкі upgrades та power-loss gates залишаються explicit pending. Provider/container composition — один owner; нові helper/test scopes не міняють format без ADR contract.
+
+### Seeded receipt lifetime correction
+
+REQ-STORAGE-014 / AC-RC-003 also maps AC-AISQL-014 and TASK-AISQL-023. Exact
+run37070004864 Windows job111047630131 passed135/136, no skips; batch7/seed1708
+cancelled during receipt write after atomic assertions and cleanup completed.
+The log cannot identify the trial, deadline source or storage damage. Move the
+unchanged receipt into the original trial try immediately after atomic/durable
+assertions, matching the Receipt-before-Cleanup diagram above. Keep original15s
+linked token, all20x50 faults/trials, actual four-slot ownership and unconditional
+cleanup. The existing catch then retains seed/trial/stage for receipt failures;
+any cleanup failure still fails qualification. Owner: gates worker, RecoveryTests.cs
+RunSeededCrashTrialAsync call placement only; root joins exact three-OS recovery
+and1000success receipts perOS. Existing ADR-035 lifetime/test contracts suffice;
+ADR:N/A for additional architecture because this is test-harness ordering only,
+with no product/wire/persistence/topology change. Rollback reintroduces the
+post-cleanup receipt cancellation risk without altering production data.