Join native request work before node shutdown

Commit all current KeyLoad changes, consume published Communication 10.2.11 and retain the owning release and original development evidence. Full Release build, formatter and governance pass. Native owner/kernel cases pass 7/7; full normal unit is 3247/3248 with the original process-identity failure retained. RF3, scalar, Linux and endurance acceptance remain open.

Author: KSemenenko

Directory.Packages.props

@@ -8,9 +8,9 @@
     <PackageVersion Include="Cartograph.Catalog" Version="0.1.0-alpha" />
     <PackageVersion Include="ZoneTree" Version="1.9.8" />
     <PackageVersion Include="ZoneTree.FullTextSearch" Version="1.0.9" />
-    <PackageVersion Include="ManagedCode.Communication" Version="10.2.9" />
-    <PackageVersion Include="ManagedCode.Communication.AspNetCore" Version="10.2.9" />
-    <PackageVersion Include="ManagedCode.Communication.Orleans" Version="10.2.9" />
+    <PackageVersion Include="ManagedCode.Communication" Version="10.2.11" />
+    <PackageVersion Include="ManagedCode.Communication.AspNetCore" Version="10.2.11" />
+    <PackageVersion Include="ManagedCode.Communication.Orleans" Version="10.2.11" />
     <PackageVersion Include="ManagedCode.TimeSeries" Version="10.0.3" />
     <PackageVersion Include="ManagedCode.Storage.FileSystem" Version="10.0.13" />
     <PackageVersion Include="ManagedCode.Orleans.Graph" Version="10.0.10" />

docs/ADR/ADR-002-command-idempotency.md

@@ -24,6 +24,38 @@ Persisting command identity with effects resolves lost responses across process
 
 Dependencies: [ADR-001](ADR-001-partition-identity-affinity.md), [ADR-003](ADR-003-durability-ack-barrier.md), [ADR-004](ADR-004-committed-read-views.md), [ADR-005](ADR-005-canonical-keyspace-codec.md), [ADR-011](ADR-011-format-upgrades.md), and [ADR-016](ADR-016-atomic-physical-placement.md). Stop if canonical fingerprint or dedup expiry behavior is not specified. Do not infer exactly-once handler execution or include an external API call in the database transaction.
 
+## TASK-DSTORE-COMMAND-100-RESTART implementation contract
+
+Current command outcomes have no automatic TTL or purge implementation. A
+matching authorized retry resolves its retained canonical outcome in the same
+store incarnation. This does not promise a finite minimum retention period,
+eternal replay after explicit store/record removal, or replay across a changed
+incarnation; an existing outcome from another incarnation is TokenInvalidated.
+Adding expiry, purge, a minimum temporal horizon or an outcome-format change
+requires its own accepted policy and migration contract before implementation.
+
+REQ-DSTORE-006 / AC-DSTORE-006 and original KL-012 require two actual CrashHost
+processes over one owned native ZoneTree store. The first commits one authorized
+document/event/queue batch, verifies exactly one hundred matching retries and
+changed-content Conflict, captures the original complete receipt as bounded test
+evidence, and waits at the existing real process-kill handshake. The parent kills
+and joins that child. A distinct second process opens the same store and verifies
+one hundred more matching retries, the retained authorized outcome, unchanged
+receipt and single effects, changed-content Conflict and healthy follow-up.
+Evidence sidecars are never canonical recovery authority. Physical commit/log
+position may advance on replay; the original receipt, domain effects and outbox
+cut must remain exact. A runner-only close/reopen is not process-restart proof,
+and process kill is not power-loss qualification.
+
+Ordered ownership: root freezes this contract and owns the minimal mode dispatch
+join in `tests/KeyLoad.CrashHost/Features/StorageRecovery/Helpers/CrashHostApplication.cs`.
+The query worker owns new scenario/helper files only under CrashHost and
+RecoveryTests `Features/DocumentStorage/`; one canonical DocumentStorage slice
+applies on both surfaces. Root reviews, joins, runs the actual Aspire recovery
+entry, retains original artifacts and updates KL-012. No new project, public API,
+serializer/fingerprint change, TTL path or copied database implementation is in
+scope. Root retains the required exact-source Linux recovery and RF3 gates.
+
 ```mermaid
 flowchart LR
     Request[Authenticated command ID and payload] --> Canonical[Resolve scope and fingerprint]

docs/ADR/ADR-018-global-rank-fusion.md

@@ -33,3 +33,27 @@ Related: `REQ-SEARCH-001/AC-SEARCH-001`, `REQ-SEARCH-002/AC-SEARCH-002`, `REQ-SE
 5. Run TUnit and RF3 Search through .NET and official MCP SDK in GitHub Actions; publish quality evidence with source SHA before claiming qualification.
 
 Current source: `src/KeyLoad.Query/Features/Search/Queries/SearchEngine.cs`, `src/KeyLoad.Core/GraphAndSeries.cs`; current tests: `tests/KeyLoad.UnitTests/GraphAndSearchTests.cs`. Planned external/provider files are owned by Search, not this ADR. Dependencies: ADR-009, ADR-019, ADR-020, ADR-022. Owner: Search lead; global quality/review join: root.
+
+## Controlled prototype stage,2026-10-05
+
+TASK-SEARCH-QUALITY-CORPUS implements REQ/AC-SEARCH-007 in
+[Search](../Features/Search.md#task-search-quality-corpus-controlled-relevance-and-window-sensitivity).
+Root freezes32 fixed documents, four-dimensional vectors, six independent query
+and graded-qrel definitions, Recall@5/10, MRR@10, nDCG@5/10 and test-only
+per-branch widths1/4/8/32. Independent eligibility and metric goldens accompany
+real ZoneTree/ZoneTree.FullTextSearch and production three-way ranking. There is
+no production per-branch window control in SearchRequest; this stage therefore
+measures sensitivity over complete real branch observations using the existing
+fusion and merger kernels, with explicit truncation. It cannot qualify an absent
+request-time feature or a distributed scorer/statistics epoch.
+
+Ordered delivery: root freezes the linked REQ/AC contract; Luna cluster_wave adds
+only new HybridQuality-prefixed Search unit cases/helpers/pure data; root reviews
+the qrels, independent oracles, actual-provider ownership and numeric limits,
+integrates and runs Aspire normal/scalar tests with original source inventories,
+then commits. Existing production source and BenchmarkComparisons ownership stay
+with their current owners. No dependency, persisted format, wire or runtime
+ranking change occurs; removal of the controlled tests is the rollback. Actual
+native RF3 clients, scorer-variant and masking/freshness/performance comparisons,
+preselected primary target and authenticated gain/no-gain publication remain later
+join points. This ADR remains Proposed for the broader global-ranking contract.

docs/ADR/ADR-082-native-cqrs-streams.md

@@ -70,6 +70,17 @@ AC-CRS-003 also hardens the existing canonical offline verifier/upgrade process
 
 TASK-CRS-C1-FATAL uses published Communication10.2.9's native CqrsRuntimeFailures.FindFatal for direct and deeply nested AggregateException failures. New test-owned RequestCqrsFatalSettlement controls exercise native creation, pulls, disposal and activation cleanup, preserve exact preconstructed fatal identity with primary/disposal/activation precedence, and join every admitted cleanup before a healthy following stream. Ordinary failure and cancellation controls remain required. Owning publication and local mechanism tests alone do not qualify C1 or mark this ADR Implemented.
 
+Checkpoint2026-10-05: TASK-CRS-CANCEL-FAILURE-JOIN is delivered through actual
+published Communication10.2.11, successful Release37250149897 and verified official
+NuGet restoration/DLL bytes. Owning native TUnit passed1363/1363. TASK-CRS-NODE-WORK-JOIN
+is integrated with actual node drain preceding native silo stop and physical
+cleanup. Aspire-selected owner/kernel tests passed7/7; full normal unit passed
+3247/3248 and remains a failed gate because the actual Node process identity test
+failed. Both source/runtime input inventories are unchanged. See the linked
+feature's original development and delivery receipts. RF3, migration and complete
+scalar/Linux/resource/endurance qualification remain open; this ADR is not marked
+Implemented by a source checkpoint or an owning package release.
+
 Rollback of C0 removes unused test infrastructure; it cannot establish product stream readiness. Later product rollout, rollback, long-work checkpoint authority, terminal cancellation and fault qualification require their concrete accepted contracts. Frontend N/A: no UI. Required real SDK/MCP Docker/Aspire RF3, recovery, resource and exact-source Linux gates remain mandatory.
 
 TASK-CRS-C1-IMAGE/RF3 follow the accepted same-epoch image/AppHost contract in NativeCqrsRequestV2. The immutable prior executable is377886f35928866f083806062b446056d64539e3, data epoch6/RPC1/peer2, independently inventoried against all native Git blobs. A distinct ClusterRouting producer/proof binds that source separately from the actual current Linux docker-rf3 job. Root adds only an explicitly enabled ephemeral fixed-three-voter test image seam and preserves homogeneous current and native5 proofs. Genuine cold seed, mixed rejection, homogeneous current rollout/receipt preservation, homogeneous rollback, and two-survivor/rejoin waves use SDK/official MCP through owned Aspire resources; live HTTP/cache/shutdown controls remain separate. Per-voter admission, exact image/reference metadata, typed public rejection, finite bounds, cleanup/join ownership, files, worker roles and evidence are frozen in that feature contract before writes.

docs/Features/ClusterRouting/NativeCqrsRequestV2.md

@@ -131,6 +131,24 @@ No public stream, binary alias/field ID, database format or dependency replaceme
 is introduced. Rollback retains the previous published package and its explicit
 failed-join evidence; no local package or project reference qualifies delivery.
 
+Delivery checkpoint2026-10-05: the owning repair is published as10.2.11 from
+Communication commit7777c90761f04b4bcffec1b5846fb74f96b8c072 through successful
+Release37250149897. Owning full native TUnit passed1363/1363. KeyLoad's three
+central references were restored from the official NuGet v3 feed; signed package
+repository commits and restored runner DLL bytes match the actual published
+payloads. The [delivery receipt](../../implementation/native-cqrs-cancel-join-delivery-2026-10-05.json)
+retains original evidence. Earlier10.2.9/10.2.10 records below remain history.
+
+NodeWork development checkpoint75g: full Release build passed with0 warnings and
+0 errors; actual Aspire selected owner/kernel tests passed7/7, including the real
+held-finally and throwing-callback join. Full normal unit passed3247/3248; its one
+failure was an actual process StartTime observation in the comparison process-tree
+test, with its cleanup cancellation retained. The full gate is failed. Both
+cohorts' HEAD, all source and executable/runtime input inventories stayed unchanged;
+only explicitly identified native TestCluster output logs are excluded from inputs.
+The [development receipt](../../implementation/node-work-development-2026-10-05.json)
+does not close the RF3 phase, migration, scalar/Linux, resource or endurance gates.
+
 TASK-CRS-C1-PRIOR-CHECKPOINT, accepted 2026-10-05, retains AC-CRS-002 and
 ADR-077/091's actual cold native6-to-native7 migration evidence after original
 run37242346547. One ten-document command does not produce the required native

docs/Features/DocumentStorage.md

@@ -39,6 +39,14 @@ The current root-level files are documented migration debt under ADR-032, not th
 | REQ-DSTORE-003: enforce row/field policy at every document boundary | AC-DSTORE-003 passes when unauthorized row writes/reads and protected field use fail or project according to policy; tenant or row ownership cannot be supplied to gain access. | Existing `NestedSensitiveFieldsAreOmittedAndAliasedPredicateAndSortAreDenied`, `RowScopeAndTenantCannotBeForged`; actual field/row/tenant matrix plus isolated replacement-write and Delete-index-use controls. Local full-suite evidence below; exact-source CI qualification pending. |
 | REQ-DSTORE-004: share an atomic transaction domain with eligible events and queues | AC-DSTORE-004 passes when document + event + local enqueue commit together or all remain absent, same command retry returns the stored outcome, and identical partition-key text in unrelated domains stays isolated. | Existing `DocumentEventAndQueueCommitTogetherAndCommandRetryDoesNotRepeatEffects`, `UniqueConflictRollsBackDocumentIndexEventAndEnqueue`, `SameLiteralPartitionKeyCannotCrossTransactionDomains`. CI qualification pending. |
 | REQ-DSTORE-005: reuse transaction-scoped document images | AC-DSTORE-005 passes when one before-record lookup supplies CRUD and outbox; no final staged lookup/decode is required; exact bytes, revisions, tombstones, indexes, authorization, quotas and sequential same-ID mutations remain intact. | TASK-MP-007I in ADR-035; new real-store paired-size/counter and before/after/failure cases plus existing transaction/change-feed/recovery/RF3 regressions; GitHub evidence pending. |
+| REQ-DSTORE-006: retain command identity and outcome atomically across real process restart | AC-DSTORE-006 passes when two distinct actual CrashHost processes execute one hundred same-ID/same-content retries each around a real first-process kill; every result matches the original complete receipt, while one document revision, one event, one Ready queue message and the exact batch outbox cut remain. Same-ID/changed-content returns Conflict without changing effects or the original outcome; a fresh authorized command succeeds afterward. | TASK-DSTORE-COMMAND-100-RESTART in ADR-002; new `CommandIdempotencyProcessRecoveryTests` and actual CrashHost scenario under `Features/DocumentStorage/`. This remains planned until original Aspire/native reports exist. |
+
+The current command-outcome retention contract has no automatic TTL/purge path.
+Retries are supported while the original outcome remains in the canonical store
+with the same incarnation and current persisted authorization. No finite minimum
+time window or retry guarantee after explicit outcome/store removal or changed
+incarnation is advertised. This is documented current behavior, not an implemented
+expiry policy. A new policy needs an accepted ADR and its own qualification.
 
 The accepted [ADR-035 document image contract](../ADR/ADR-035-memory-performance.md)
 assigns CRUD handlers and new matching tests to one worker, and shared

docs/Features/ResourceExecution.md

@@ -477,3 +477,32 @@ measurement claim follows from authored counters or local static validation.
 Рішення: [ADR-010](../ADR/ADR-010-query-budgets-security.md), [ADR-028 scheduling time](../ADR/ADR-028-persisted-scheduling-time.md), [ADR-031 modular resource isolation](../ADR/ADR-031-modular-all-in-one-resource-isolation.md), ADR-035. TimeProvider.System залишається host/Orleans clock; operation clock не заморожує cluster runtime. Mixed query/search/graph/event/queue workloads потребують real correctness+liveness tests, а не окремих synthetic doubles.
 
 Shared primitive/config/telemetry composition мають одного integration owner; бізнес-work і matching tests залишаються в owning `Features/<SliceName>/`. Configuration validation, invalid bounds, zero/empty work, overflow, cancellation/deadline і healthy follow-up входять до AC. Freeze shared limits → real regression source → feature fixes → integrated GitHub gates → actual measurements. Local doc review не доводить numeric coverage, memory ceilings чи performance gain; metrics публікуються тільки з successful GitHub JSON.
+
+### TASK-CACHE-EXPIRY-ORACLE: real prepare-origin expiry measurement
+
+REQ-CACHE-EXPIRY-ORACLE-001 preserves REQ-CACHE-004/005 and the existing
+15-second prepare-origin lease and 10-second acceptance ceiling. A real-clock
+test must distinguish prepare age from receipt age while observing the actual
+permit and native ZoneTree cache. Source clocks, production bounds and persisted
+authority are unchanged. ADR-058 already specifies this contract; no new boundary
+or timing semantics are introduced.
+
+AC-CACHE-EXPIRY-ORACLE-001 maps to
+`CacheReadPermitExpiryTests.LeaseExpiresAtFifteenSecondsFromItsPrepareTimestamp`
+and `ZoneTreeCoordinatedPointCacheReadTests.ExpiredWarmReceiptFallsBackToOneNativeReadWithIdenticalBytes`.
+Only these two methods are isolated with TUnit `NotInParallel`. Accept at a
+measured prepare age of at least six seconds and below the unchanged ten-second
+ceiling; use monotonic elapsed-time polling with a one-millisecond minimum delay.
+Observe after prepare age reaches fifteen seconds plus the existing fifty-
+millisecond boundary margin, and still require receipt age below fifteen seconds.
+Keep the original thirty-second caller bound, stale capture/revision rejection,
+renewal proof, identical bytes and exactly one native fallback lookup. Open, seed
+and compose the real ZoneTree store before starting its lease measurement.
+Neither a fake clock nor an increased lease/deadline or removed assertion is an
+acceptable correction. Preserve the two failed original Linux reports from
+run 37249197752 before recording any new local or exact-source Linux result.
+
+The lead freezes and reviews this scope, joins the source and owns Aspire tests,
+receipts and commits. The query worker owns a private patch for only the two
+existing Cases files and, if required, one cohesive real-clock Helpers file.
+Other test methods, cache admission and production code are outside this stage.