Verify TimeProvider clocks and deterministic storage budgets

Author: KSemenenko

README.md

@@ -378,8 +378,8 @@ For detailed status, see the [implementation tracker](docs/implementation/status
 
 The [TimeProvider migration](docs/Features/ResourceExecution/TimeProvider.md)
 supplies explicit clocks for timestamps, elapsed budgets and managed timers.
-Its local Aspire analyzer suite passes 388 tests; controlled real-operation
-test sources are joined, while full build and runtime qualification remain open.
+Controlled-time regressions and the remaining runtime qualification gates
+are tracked in the feature specification.
 
 Website publication runs independently in CI when source changes. It uses the newest completed benchmark run with a verified aggregate when available; otherwise it publishes the product site without performance figures. A completed benchmark run triggers a fresh website build. [ADR-112](docs/ADR/ADR-112-independent-website-publication.md) records the source, artifact and publication checks; the revised route still needs delivered-source Linux CI and Pages verification.
 

docs/Features/ResourceExecution/TimeProvider.md

@@ -1,6 +1,6 @@
 # ResourceExecution: controlled time
 
-Status: source joined; full build and runtime qualification remain incomplete. Local development verification is distinct from Linux RF3 qualification.
+Status: source joined; local complete build, formatter and focused runtime tests passed. Full runtime qualification remains incomplete. Local development verification is distinct from Linux RF3 qualification.
 Decision: [ADR-115](../../ADR/ADR-115-time-provider.md).
 
 ## Goal and scope
@@ -30,9 +30,9 @@ Validation skills in order: quality-ci (check native gates), analyzer-config (ex
 
 ## Development evidence, 2026-10-06
 
-The [development receipt](../../implementation/time-provider-development-2026-10-06.json) records actual commands, native report hash and unresolved gates. The complete Aspire-owned analyzer suite passed 388/388 tests, including readonly Stopwatch fields and the controlled migration deadline. Product clock modules, ComparisonHost and the final integration test sources compile with analyzers. The complete solution and canonical formatter remain blocked by parallel BackupRestore/cancellation-flow diagnostics; process recovery stops during prior-source preparation. The local RF3 suite failed (16 passed, 142 failed) on cluster-profile reparse paths and unavailable prior-server proof. These are local macOS development observations, with no Linux qualification, coverage, CRAP or performance claim.
+The [development receipt](../../implementation/time-provider-development-2026-10-06.json) records actual commands, native report hashes and unresolved gates. The complete Aspire-owned analyzer suite passed 388/388 tests, including readonly Stopwatch fields and the controlled migration deadline. The final complete Release solution build passed with zero warnings/errors, and the canonical formatter passed after the storage-policy clock fixture edit. Eight new controlled-time operation cases, three existing read-budget cases and six storage-policy cases passed through Aspire. A duplicate whole-unit attempt was gracefully stopped while another shared whole-unit run remained active; its shutdown exit zero is not a test pass. The full unit gate still has startup, CLI/process and source-inventory failures without a completed passing report. Process recovery stops during prior-source preparation. The local RF3 suite failed (16 passed, 142 failed) on cluster-profile reparse paths and unavailable prior-server proof. These are local macOS development observations, with no Linux qualification, coverage, CRAP or performance claim.
 
-Controlled real-operation sources: `TimeProviderOperationTests` (JSON/native command timestamp and retry, query/due-work expiry, UTC versus monotonic budget), `NativeReadCutClockTests` (real snapshot expiry, released slot and preserved bytes), and `DueWaitControlledClockTests` (real apply waiter fallback/cancellation and joined timers). Fresh UnitTests compilation and execution remain required. The controlled timer supports these single-threaded one-shot workflows; it is not qualified as a general concurrent or periodic timer implementation. Framework-native synchronous process/thread waits without a provider overload remain bounded OS observations; managed delays and timed task waits use provider overloads.
+Passing real-operation workflows: `TimeProviderOperationTests` (5 cases: JSON/native command timestamp and retry, query/due-work expiry, UTC versus monotonic budget), `NativeReadCutClockTests` (1 case: real snapshot expiry, released slot and preserved bytes), `DueWaitControlledClockTests` (2 cases: real apply waiter fallback/cancellation and joined timers), `ReadExecutionTests` (3 existing cases), and `ZoneTreeStorageExecutionOptionsTests` (6 cases, including deterministic record-budget assertions under an unchanged one-second elapsed policy). Full unit/scalar and recovery/RF3 gates remain required. A native `/private/tmp` directory avoids the macOS system-temp alias without weakening filesystem guards. The controlled timer supports these single-threaded one-shot workflows; it is not qualified as a general concurrent or periodic timer implementation. Framework-native synchronous process/thread waits without a provider overload remain bounded OS observations; managed delays and timed task waits use provider overloads.
 
 ```mermaid
 flowchart LR

docs/implementation/status.json

@@ -5,18 +5,18 @@
     "feature": "docs/Features/ResourceExecution/TimeProvider.md",
     "decision": "docs/ADR/ADR-115-time-provider.md",
     "sourceStage": "clock_migration_and_deterministic_real_operation_regressions_joined",
-    "canonicalBuildPassed": false,
-    "canonicalFormatterPassed": false,
+    "canonicalBuildPassed": true,
+    "canonicalFormatterPassed": true,
     "runtimeQualified": false,
     "fullTaskComplete": false,
-    "verificationStage": "Aspire_analyzers_388_passed_shared_build_and_runtime_gates_blocked",
+    "verificationStage": "local_canonical_build_and_format_passed_Aspire_analyzers388_and_focused_operation_tests17_passed_runtime_qualification_open",
     "analyzerTestsPassed": 388,
     "localDevelopmentReceipt": "docs/implementation/time-provider-development-2026-10-06.json",
     "remaining": [
-      "Fresh passing complete solution build and canonical formatting",
-      "Execute controlled engine/read-cut/replica tests through freshly compiled Aspire unit runner",
-      "Complete unit/scalar/recovery/RF3 and exact-source Linux gates"
-    ]
+      "Complete unit/scalar/recovery/RF3 and exact-source Linux gates",
+      "Functional coverage and source-bound CRAP remain unmeasured"
+    ],
+    "focusedOperationTestsPassed": 17
   },
   "ORLEANS-PRIMITIVES-001": {
     "title": "Native Orleans capability review, services, messaging, transactions and jobs",

docs/implementation/time-provider-development-2026-10-06.json

@@ -3,10 +3,10 @@
   "scope": "TimeProvider local development; not Linux qualification",
   "feature": "docs/Features/ResourceExecution/TimeProvider.md",
   "decision": "docs/ADR/ADR-115-time-provider.md",
-  "sourceRevisionObserved": "51ce19cfbb086b316d41cd959e98946d78cc26e3",
+  "sourceRevisionObserved": "7e218a1f8e161d1107457f5b035759b8af2cafdd",
   "concurrentSourceChangesExcluded": true,
-  "canonicalBuildPassed": false,
-  "canonicalFormatterPassed": false,
+  "canonicalBuildPassed": true,
+  "canonicalFormatterPassed": true,
   "runtimeQualified": false,
   "fullTaskComplete": false,
   "analyzers": {
@@ -67,8 +67,8 @@
     ]
   },
   "builds": {
-    "canonical": "failed: 33 parallel BackupRestore artifact diagnostics and one AppHost signal join; join subsequently repaired and scoped AppHost build passed",
-    "canonicalLog": "/tmp/keyload-time-solution-build-final.log",
+    "canonical": "passed complete Release solution build after final storage-policy controlled-clock fixture edit: 0 warnings/errors; 6m22.24s; local shared checkout with unrelated source changes",
+    "canonicalLog": "/tmp/keyload-time-solution-after-readcut-fixture.log",
     "compiledModules": [
       "Core",
       "Query",
@@ -87,13 +87,29 @@
       "Server",
       "Analyzers.Tests",
       "CrashHost",
-      "RecoveryTests"
+      "RecoveryTests",
+      "AppHost",
+      "IntegrationTests",
+      "ComparisonTests",
+      "Artifacts",
+      "Cli",
+      "UnitTests",
+      "SiteTests"
     ],
     "scopedAppHost": "passed 0 warnings/errors, current product references; no replacement for solution gate",
-    "unitSource": "blocked by 8 parallel cancellation-flow and BackupRestore diagnostics; controlled operation cases not executed",
-    "unitLog": "/tmp/keyload-time-unit-source-final-r2.log",
+    "unitSource": "passed in complete solution build, including final controlled operation fixtures",
+    "unitLog": "/tmp/keyload-time-solution-after-readcut-fixture.log",
     "integrationSource": "passed 0 warnings/errors after async deadline ownership and disposal join repair",
-    "integrationLog": "/private/tmp/keyload-time-runtime-integration-source-build.log"
+    "integrationLog": "/private/tmp/keyload-time-runtime-integration-source-build.log",
+    "comparisonTestsSource": "passed 0 warnings/errors; current provider-taking Redis fixture overload also compiled in final canonical attempt",
+    "comparisonTestsLog": "/tmp/keyload-time-comparison-tests-final.log",
+    "lastIntermediateCanonical": "failed26 errors:24 parallel BackupRestore diagnostics and2 Redis joins, latter subsequently repaired",
+    "lastIntermediateCanonicalLog": "/tmp/keyload-time-solution-after-ownership-fix.log",
+    "canonicalErrors": 0,
+    "canonicalWarnings": 0,
+    "canonicalCommand": "dotnet build KeyLoad.slnx --no-restore --configuration Release --disable-build-servers -m:1",
+    "postReadCutFixtureVerification": "passed complete solution build and canonical formatter after final controlled-clock storage fixture edit",
+    "lastIntermediateCanonicalPassedLog": "/tmp/keyload-time-solution-latest.log"
   },
   "recovery": {
     "command": "dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=recovery",
@@ -103,17 +119,17 @@
   },
   "format": {
     "command": "dotnet format KeyLoad.slnx --verify-no-changes --no-restore",
-    "exitCode": 2,
-    "status": "parallel BackupRestore and cancellation-flow diagnostics; scoped clock formatting passed",
-    "log": "/tmp/keyload-time-format-final.log"
+    "exitCode": 0,
+    "status": "passed canonical verifier after final controlled-clock storage fixture edit",
+    "log": "/tmp/keyload-time-format-after-readcut-fixture.log",
+    "lastIntermediatePassedLog": "/tmp/keyload-time-format-post-repairs.log"
   },
   "governance": "passed 27 projects and 4 modules",
   "performance": "unmeasured; no acceleration claim",
   "coverageAndCrap": "unmeasured",
   "remaining": [
-    "Fresh passing complete solution build and canonical formatting",
-    "Execute controlled engine/read-cut/replica tests through freshly compiled Aspire unit runner",
-    "Complete unit/scalar/recovery/RF3 and exact-source Linux gates"
+    "Complete unit/scalar/recovery/RF3 and exact-source Linux gates",
+    "Functional coverage and source-bound CRAP remain unmeasured"
   ],
   "rf3": {
     "command": "dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=rf3",
@@ -131,5 +147,121 @@
     "nativeReportSha256": "98fa6f4065dab9fa6fd13af1dd0232305ffcdc5cf1d204f9d16876bd40b9e54d",
     "status": "failed local development: cluster profile rejects macOS reparse temp paths; genuine prior-server proof unavailable; no passing Docker RF3 claim",
     "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T083253_a37d4221.log"
+  },
+  "controlledOperationTests": [
+    {
+      "className": "TimeProviderOperationTests",
+      "command": "TMPDIR=/private/tmp dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/TimeProviderOperationTests/*' --KeyLoadTests:ResultsDirectory=TestResults/time-provider-engine-final --KeyLoadTests:ReportTrx=true",
+      "exitCode": 0,
+      "summary": {
+        "total": 5,
+        "passed": 5,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "nativeReport": "TestResults/time-provider-engine-final/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+      "nativeReportSha256": "f68d4a18103727b0d00ec835c5f02fc0de30896b1298df7eaf4b0cb171414fd3",
+      "publicationGeneration": "50eca46d107a47fa979b219f0a42c4a2",
+      "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T085629_5dc2449b.log"
+    },
+    {
+      "className": "NativeReadCutClockTests",
+      "command": "dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/NativeReadCutClockTests/*' --KeyLoadTests:ResultsDirectory=TestResults/time-provider-read-cut --KeyLoadTests:ReportTrx=true",
+      "exitCode": 0,
+      "summary": {
+        "total": 1,
+        "passed": 1,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "nativeReport": "TestResults/time-provider-read-cut/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+      "nativeReportSha256": "4a760d73666ac0271aa4d5d8452ce7f3c95a68c7727af0ae5d39c345553ffd3d",
+      "publicationGeneration": "d54fcca10db64f859f4436e5c0bfc312",
+      "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T084815_8cf7b840.log"
+    },
+    {
+      "className": "DueWaitControlledClockTests",
+      "command": "TMPDIR=/private/tmp dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/DueWaitControlledClockTests/*' --KeyLoadTests:ResultsDirectory=TestResults/time-provider-due-wait-final --KeyLoadTests:ReportTrx=true",
+      "exitCode": 0,
+      "summary": {
+        "total": 2,
+        "passed": 2,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "nativeReport": "TestResults/time-provider-due-wait-final/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+      "nativeReportSha256": "f12f68246c9bd0e47836531c987cbda2119bc124743df10d3a8c2cc1cec4cf56",
+      "publicationGeneration": "5290d16876d14fe09a133251d03ec640",
+      "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T085657_fda40946.log"
+    },
+    {
+      "className": "ReadExecutionTests",
+      "command": "TMPDIR=/private/tmp dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ReadExecutionTests/*' --KeyLoadTests:ResultsDirectory=TestResults/time-provider-read-execution --KeyLoadTests:ReportTrx=true",
+      "exitCode": 0,
+      "summary": {
+        "total": 3,
+        "passed": 3,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "nativeReport": "TestResults/time-provider-read-execution/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+      "nativeReportSha256": "dd268415510a88abda207e1afad9ae4eef2ab8a9d51bf6acfa7cc28e61a9e4f0",
+      "publicationGeneration": "c6928f27f23047e19a737d6694d0f555",
+      "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T085721_1e1b70d9.log"
+    },
+    {
+      "className": "ZoneTreeStorageExecutionOptionsTests",
+      "command": "TMPDIR=/private/tmp ASPIRE_SUPPRESS_CLI_RUN_HOOK=true dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ZoneTreeStorageExecutionOptionsTests/*' --KeyLoadTests:ResultsDirectory=TestResults/time-provider-storage-policy-final --KeyLoadTests:ReportTrx=true",
+      "exitCode": 0,
+      "summary": {
+        "total": 6,
+        "passed": 6,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "nativeReport": "TestResults/time-provider-storage-policy-final/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+      "nativeReportSha256": "66558f2ffa0afb61954cb835f42a4aac58f8cad81f5c10be7dc308bdfed29201",
+      "publicationGeneration": "e9d624d8386849a9816b4232139df344",
+      "log": "/tmp/keyload-time-storage-policy-final-run.log",
+      "entryPoint": "native Aspire AppHost with its SDK CLI run hook suppressed; actual AppHost still owns runner readiness, execution, exit and shutdown; isolated storage-only cases avoid replacing the other full-suite AppHost"
+    }
+  ],
+  "controlledOperationTestsNote": "8 new controlled-clock cases, 3 existing read-budget cases and 6 storage-policy cases passed. The storage record-budget case now uses the controlled clock while keeping its 1-second policy and real store. First query fixture needed explicit full-scan consent; cancellation fixture joins real apply before asserting state. Native temp avoids macOS system-temp aliases without changing filesystem guards.",
+  "fullUnitAttempt": {
+    "command": "TMPDIR=/private/tmp dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit --KeyLoadTests:ResultsDirectory=TestResults/time-provider-unit-full --KeyLoadTests:ReportTrx=true",
+    "cliExitCode": 0,
+    "status": "interrupted deliberately with native Aspire SIGINT because a second shared whole-unit run was active; CLI shutdown exit0 is not a test pass; every owned host/test PID was observed terminated",
+    "log": "/Users/ksemenenko/.aspire/logs/cli_20261006T085852_7183a28a.log",
+    "observedFailedCasesBeforeInterruption": 11,
+    "observedFailureAreas": [
+      "CLI output retention",
+      "native benchmark child settlement",
+      "coverage CLI workflow",
+      "Orleans test-cluster startup cancellation",
+      "wall-time storage budget fixture now changed to injected controlled clock"
+    ],
+    "nativeSummaryAvailable": false,
+    "nativeCompleteReportAvailable": false
+  },
+  "focusedOperationTestsPassed": 17,
+  "fullUnitSharedFollowup": {
+    "status": "the other authorized chat owns a complete unit run; it has observed failures and source-inventory drift; no completed passing report was available at this checkpoint",
+    "resultsDirectory": "TestResults/general-options-unit-v34",
+    "log": "/private/tmp/keyload-general-options-aspire-unit-v34.log"
   }
 }