Repair SQL regression source compilation and RF3 caller lifetimes

Author: KSemenenko

ai-database-sql.plan.md

@@ -26,6 +26,9 @@ no reliable cheaper-tier correctness/routing evidence is available in this run.
 | TASK-AISQL-011; exact baseline finding and Accepted ADR-036 contract | SQL worker; OrleansNode.cs, NEW Server/Features/ClusterRouting/OrleansRpcFailure.cs and UnitTests/Features/ClusterRouting/OrleansRpcFailureTests.cs only; root owns trusted gateway/auth caller joins and docs | 011,009 | Native-exception regression sources, narrow initial-RPC classification, no retry/suppression/domain recoding; root reviews then exact-SHA build/unit/RF3 catch-up proof |
 | TASK-AISQL-012A; Accepted ADR-054 integration repair | models worker; Client/Features/BlobStorage/BlobClient.cs and NEW UnitTests/Features/BlobStorage/BlobSdkArgumentTests.cs only; root owns Send access join | 012,006,009 | Tests-first public null guards, source-compatible feature extension methods, same transport/IDs; root review + exact build/RF3 |
 | TASK-AISQL-012B; Accepted ADR-054 integration repair | gates worker; QueryCandidateReader.cs only | 008,009 | Remove depth4 nesting without new allocations/semantic changes; existing25 real-store regression cases + exact build |
+| TASK-AISQL-015; exact UnitTests compiler diagnostics | models worker; RelationalLinkageTests.cs, RelationalTestData.cs and RelationalRowTests.cs only | 002–004,009 | Correct actual ordered TUnit collection comparison and cache CompositeFormat without altering exact raw decimal fixtures or assertions; exact-SHA compiler/units join |
+| TASK-AISQL-016; exact UnitTests compiler diagnostics | SQL worker; OrleansRpcFailureTests.cs, SqlOperationBudgetTests.cs and SqlOperationCompilerTests.cs only | 005,007,011,009 | Await native CancelAsync before cancellation assertions and remove imports proven redundant by actual compiler; same errors and test coverage, no suppression |
+| TASK-AISQL-014; exact RF3 compiler diagnostics | gates worker; SqlRf3AdmissionTests.cs only, root owns static scenario member | 004,007,009 | Isolate existing per-node using/await-using ownership in VerifyNodeAsync outside fixture try/finally/loop analysis; start caller deadline after separately bounded fixture readiness; preserve all3 real-node assertions, same-ID control progress and schema/linkage contracts; no suppression/test weakening |
 
 No worker may expand scope, change contracts/policy, install tools/dependencies,
 stash, weaken diagnostics/assertions, test locally or commit/push. Escalate any
@@ -107,7 +110,7 @@ build/format/governance/unit/scalar/recovery and analyzer jobs passed. Required
 runtime failures remain:
 
 - [ ] RetainedReplicaCatchesUpThroughNativeSnapshotAndKeepsCommandOutcomes:
-  [RF3 job111033735266](https://github.com/managedcode/KeyLoad/actions/runs/37065200835/job/111033735266),45/46 passing, no skips. RecoveryRequired after stopped replica; actual node1 logs show distributed-directory/placement connection rejection to node2. Inspect transport failure classification; do not broaden accepted retry errors to hide genuine recovery damage.
+  [RF3 job111033735266](https://github.com/managedcode/KeyLoad/actions/runs/37065200835/job/111033735266),45/46 passing, no skips. RecoveryRequired after stopped replica; node1 logs also contain distributed-directory/placement connection rejection to node2, but the exact failed request dispatch phase and native exception class were not captured. The narrow native RPC repair is a contract correction, not proof of that failure's root cause; qualify the unchanged catch-up test and preserve genuine recovery damage.
 - [ ] AspireRunsIdenticalScenariosAgainstRealRf3AndExternalEngines:
   [comparison job111033735452](https://github.com/managedcode/KeyLoad/actions/runs/37065200835/job/111033735452), resource Waiting with no creation/health; lifecycle timed out. Concurrent comparison work is outside this scoped source commit.
 - [ ] AspireTimeseriesProfileEmitsOracleReportAndProtectsForeignSchema:
@@ -157,3 +160,19 @@ reached Server build after those repairs and stopped on one unnecessary import:
   settings into the dedicated fixture so SDK/MCP SQL rejection and direct-control
   progress can be deterministic. Concurrent occupied-lane saturation stays a
   separate qualifier; AC007 stays open until actual GitHub runtime evidence.
+
+Candidate9f3acf5b9af39ad5d8f818039b22eb8ce1b0463e / [37069574976](https://github.com/managedcode/KeyLoad/actions/runs/37069574976) repeats the full gates after the IDE0005 repair. Outcome is pending; no runtime result is inferred from source review.
+
+RF3 build job111045550869 reached IntegrationTests and exposed:
+
+- [ ] CA1822 RelationalSqlRf3Scenario.Space: make the constant-only helper static; linked graph/vector semantics remain identical.
+- [ ] CA2000 SqlRf3AdmissionTests caller allocation: the committed loop already uses using/await-using, but analyzer reports both declarations within fixture try/finally. Extract an explicit per-node disposal scope into VerifyNodeAsync; retain unconditional disposal, all three node assertions and original control semantics. Start the caller deadline after the independent bounded fixture initialization so cold startup cannot consume it. Repeat exact-SHA build and real RF3 tests.
+
+The same candidate Ubuntu job111045551116 exposed additional unit source failures before any units could execute:
+
+- [ ] CS0103 RelationalLinkageTests.CollectionOrdering: bind the actual ordered TUnit comparison API; preserve sequence equality.
+- [ ] CA1863 RelationalTestData/RelationalRowTests string.Format: cache CompositeFormat preserving byte-for-byte numeric JSON fixture contents.
+- [ ] CA1849 OrleansRpcFailureTests/SqlOperationBudgetTests cancellation: await native CancelAsync before exercising the pre-cancelled boundary.
+- [ ] IDE0005 unused UnitTests imports in OrleansRpcFailureTests/SqlOperationCompilerTests: remove only compiler-identified redundant imports.
+
+TASK-AISQL-014/015/016 are complete source artifacts, reviewed and joined by root. Per-node disposal extraction, static vector-space helper, actual TUnit namespace, cached invariant formatting, awaited cancellation and unused-import removals retain every contract/assertion. Exact-SHA build/tests remain pending.

docs/implementation/central-sql.md

@@ -48,7 +48,9 @@ Required baseline dispatched after planning:
 was cancelled before any job executed. The original37065200835 later completed
 failure: all three OS build/format/governance/unit/scalar/recovery jobs and analyzer
 rules passed; RF3 passed45/46 and comparisons2/4. The remaining snapshot/catch-up
-failure shows Orleans directory/placement connection rejection to a stopped node.
+failure returned a generic recovery error. Node logs also show Orleans directory/placement
+connection rejection to a stopped node; the request dispatch phase and exact exception
+were not retained, so a causal attribution remains unproven.
 Comparisons lifecycle remained Waiting, and two retained StreamAppend smoke cases
 failed with unsupported ReadEventAsync. The root plan tracks exact jobs and cases.
 Exact new-source qualification is pending; no test count or speed result is
@@ -91,3 +93,7 @@ initial native Orleans/timeout RPC faults map to read OwnershipLost or uncertain
 command UnknownWriteOutcome. Genuine domain RecoveryRequired and caller
 cancellation retain their contracts; no extra dispatch/retry is introduced.
 Independent join review is complete; the next exact-SHA CI is the authority.
+
+Candidateaf9e0d16b / [37069241980](https://github.com/managedcode/KeyLoad/actions/runs/37069241980) stopped on one redundant namespace import before product tests. Candidate9f3acf5b9 / [37069574976](https://github.com/managedcode/KeyLoad/actions/runs/37069574976) removes it and repeats complete qualification; results remain pending.
+
+Candidate9f3acf5b9 passed Server/ComparisonTests compilation and reached comparison tests, but IntegrationTests/UnitTests compilation failed on CA1822/CA2000, missing TUnit enum namespace, repeated-format caching, synchronous cancellation and redundant imports. Bounded source repairs preserve the existing assertions and are awaiting a fresh complete CI; RF3 and units did not execute in that candidate.

docs/implementation/status.json

@@ -1925,12 +1925,26 @@
         "run": 37068458582,
         "conclusion": "failure",
         "scope": "build_CA1062_blob_null_KLD0031_client_aggregate_KLD0033_query_nesting_no_product_runtime_execution"
+      },
+      {
+        "sha": "af9e0d16b6fdc781f7d4b9b54cd56fec152cfc56",
+        "run": 37069241980,
+        "conclusion": "failure",
+        "scope": "build_IDE0005_redundant_Server_import_no_product_runtime_execution"
+      },
+      {
+        "sha": "9f3acf5b9af39ad5d8f818039b22eb8ce1b0463e",
+        "run": 37069574976,
+        "conclusion": "in_progress",
+        "scope": "Server_ComparisonTests_build_pass_Unit_Integration_compile_failed_no_unit_RF3_execution_comparisons_pending"
       }
     ],
     "sourceRepairsAwaitingQualification": [
       "feature_owned_typed_blob_extensions_shared_transport_null_guards",
       "symmetric_native_equality_early_iterator_exits",
-      "initial_native_Orleans_RPC_read_ownership_write_unknown_outcome_mapping"
+      "initial_native_Orleans_RPC_read_ownership_write_unknown_outcome_mapping",
+      "per_node_RF3_caller_disposal_and_deadline_after_readiness",
+      "test_static_TUnit_namespace_cached_format_native_async_cancel_unused_import_repairs"
     ],
     "evidence": [
       "docs/Features/QueryExecution.md",

tests/KeyLoad.IntegrationTests/Features/QueryExecution/SqlRf3AdmissionTests.cs

@@ -24,25 +24,31 @@ public async Task AcAisql007InsufficientDataBytesRejectSdkAndOfficialSqlBeforeId
     {
         var fixture = new ClusterFixture(new HttpAdmissionLimits
         { MaxReservedBytes = DataBytes, HeavyReadReservedBytes = HeavyReadBytes });
-        using var deadline = McpCallerDeadline.Create();
         try
         {
             await fixture.InitializeAsync();
+            using var deadline = McpCallerDeadline.Create();
             var partition = new PartitionRef(TenantPrefix + Guid.NewGuid().ToString(McpCallerProtocol.GuidFormat),
                 Database, Domain, Guid.NewGuid().ToString(McpCallerProtocol.GuidFormat));
             foreach (var node in new[] { McpCallerProtocol.Node1, McpCallerProtocol.Node2, McpCallerProtocol.Node3 })
             {
-                using var http = McpCallerHttp.Create(fixture, node);
-                var sdk = new KeyLoadClient(http, fixture.AdminKey);
-                await using var mcp = await McpOfficialClient.ConnectAsync(fixture, node, fixture.AdminKey, deadline.Token);
-                await VerifySdkRejectionAsync(http, sdk, fixture.AdminKey, partition, deadline.Token);
-                await VerifyOfficialRejectionAsync(mcp, partition, deadline.Token);
-                await VerifyHealthyAsync(sdk, deadline.Token);
+                await VerifyNodeAsync(fixture, node, partition, deadline.Token);
             }
         }
         finally { await fixture.DisposeAsync(); }
     }
 
+    private static async Task VerifyNodeAsync(ClusterFixture fixture, string node, PartitionRef partition,
+        CancellationToken cancellationToken)
+    {
+        using var http = McpCallerHttp.Create(fixture, node);
+        var sdk = new KeyLoadClient(http, fixture.AdminKey);
+        await using var mcp = await McpOfficialClient.ConnectAsync(fixture, node, fixture.AdminKey, cancellationToken);
+        await VerifySdkRejectionAsync(http, sdk, fixture.AdminKey, partition, cancellationToken);
+        await VerifyOfficialRejectionAsync(mcp, partition, cancellationToken);
+        await VerifyHealthyAsync(sdk, cancellationToken);
+    }
+
     private static async Task VerifySdkRejectionAsync(HttpClient http, KeyLoadClient sdk, string administratorKey,
         PartitionRef partition, CancellationToken cancellationToken)
     {

tests/KeyLoad.IntegrationTests/Features/RelationalStorage/RelationalSqlRf3Scenario.cs

@@ -10,7 +10,7 @@ internal sealed record RelationalSqlRf3Scenario(PartitionRef Partition, Resource
     internal EntityRef First => new(Partition, RelationalSqlRf3Tokens.Table, RelationalSqlRf3Tokens.FirstId);
     internal EntityRef Second => new(Partition, RelationalSqlRf3Tokens.Table, RelationalSqlRf3Tokens.SecondId);
     internal StreamRef Stream => new(Partition, RelationalSqlRf3Tokens.Streams, RelationalSqlRf3Tokens.StreamId);
-    internal VectorSpace Space => new(RelationalSqlRf3Tokens.VectorSpace, RelationalSqlRf3Tokens.Dimension,
+    private static VectorSpace Space => new(RelationalSqlRf3Tokens.VectorSpace, RelationalSqlRf3Tokens.Dimension,
         DistanceMetric.DotProduct, RelationalSqlRf3Tokens.VectorModel, RelationalSqlRf3Tokens.VectorVersion);
 
     internal static async Task<RelationalSqlRf3Scenario> CreateAsync(KeyLoadClient administrator, CancellationToken cancellationToken)

tests/KeyLoad.UnitTests/Features/ClusterRouting/OrleansRpcFailureTests.cs

@@ -2,7 +2,6 @@
 using KeyLoad.Server;
 using KeyLoad.UnitTests.Features.TestInfrastructure;
 using Microsoft.Extensions.Logging;
-using global::Orleans.Runtime;
 using global::Orleans.Runtime.Messaging;
 
 namespace KeyLoad.UnitTests.Features.ClusterRouting;
@@ -66,7 +65,7 @@ public async Task AcRoute009CancellationDomainRecoveryAndOtherFailuresRemainOuts
     public async Task AcRoute009CallerCancellationWinsOverANativeFailureAndProducesNoFailureLog(bool command)
     {
         using var cancellation = new CancellationTokenSource();
-        cancellation.Cancel();
+        await cancellation.CancelAsync();
         using var capture = new EventSourceLogCapture();
         using var factory = LoggerFactory.Create(builder => builder.AddEventSourceLogger());
         var logger = factory.CreateLogger(nameof(GrainFailureDiagnosticsTests));

tests/KeyLoad.UnitTests/Features/QueryExecution/SqlOperationBudgetTests.cs

@@ -60,7 +60,7 @@ await SqlOperationTestData.Reject(request, ErrorCode.BudgetExceeded,
     public async Task AcAiSql007CancellationWinsBeforeMalformedInputOrPayloadAllocation()
     {
         using var cancellation = new CancellationTokenSource();
-        cancellation.Cancel();
+        await cancellation.CancelAsync();
         var request = SqlOperationTestData.Call(McpCatalogExpectations.QueryCapabilities) with { Sql = null! };
         var failure = Assert.ThrowsExactly<OperationCanceledException>(() =>
             SqlOperationTestData.Compile(request, cancellationToken: cancellation.Token));

tests/KeyLoad.UnitTests/Features/QueryExecution/SqlOperationCompilerTests.cs

@@ -1,5 +1,4 @@
 using System.Text.Json;
-using KeyLoad.Query;
 using KeyLoad.Server;
 using KeyLoad.UnitTests.Features.ClientApi;
 

tests/KeyLoad.UnitTests/Features/RelationalStorage/RelationalLinkageTests.cs

@@ -1,4 +1,5 @@
 using KeyLoad.Query;
+using TUnit.Assertions.Enums;
 
 namespace KeyLoad.UnitTests.Features.RelationalStorage;
 

tests/KeyLoad.UnitTests/Features/RelationalStorage/RelationalRowTests.cs

@@ -1,4 +1,5 @@
 using System.Globalization;
+using System.Text;
 using System.Text.Json;
 
 namespace KeyLoad.UnitTests.Features.RelationalStorage;
@@ -29,6 +30,7 @@ internal sealed class RelationalRowTests
     private const string DecimalMaximum = "79228162514264337593543950335";
     private const string DecimalMinimumQuantum = "0.0000000000000000000000000001";
     private const string ValidTrailingZeros = "1.25000000000000000000000000000";
+    private static readonly CompositeFormat ParsedPropertyFormat = CompositeFormat.Parse(PropertyFormat);
 
     [Test]
     public async Task AcAisql003ValidScalarLimitsUtcAndNullableAbsenceCommitWithoutCoercion()
@@ -113,8 +115,8 @@ public async Task AcAisql003ExactNumericBoundarySpellingsRemainAccepted(string i
     private static string Replace(string row, string column, string value)
     {
         using var document = JsonDocument.Parse(row);
-        var before = string.Format(CultureInfo.InvariantCulture, PropertyFormat, column, document.RootElement.GetProperty(column).GetRawText());
-        var after = string.Format(CultureInfo.InvariantCulture, PropertyFormat, column, value);
+        var before = string.Format(CultureInfo.InvariantCulture, ParsedPropertyFormat, column, document.RootElement.GetProperty(column).GetRawText());
+        var after = string.Format(CultureInfo.InvariantCulture, ParsedPropertyFormat, column, value);
         return row.Replace(before, after, StringComparison.Ordinal);
     }
 }