Separate benchmark contracts from functional Build and Tests

Author: KSemenenko

.github/workflows/AGENTS.md

@@ -1,5 +1,9 @@
 # GitHub Actions workflows
 
+## Build and Tests scope, owner correction 2026-10-06
+
+- Build and Tests executes only solution builds, repository checks and KeyLoad functional tests. Benchmark/comparison cases and their progress, entry, counter and image-preparation checks belong exclusively to Benchmarks. Keep benchmark contracts in KeyLoad.ComparisonTests, not KeyLoad.UnitTests; retain their assertions and execution in Benchmarks. RF3 functional qualification builds only the KeyLoad server image, never a benchmark load-generator image. This explicit owner correction supersedes earlier benchmark-correctness placement in ordinary CI.
+
 ## Purpose and entry points
 - Owns repository CI qualification and GitHub Pages publication workflows.
 - Canonical workflows: `ci.yml` (restore, Release build, TUnit unit/integration/recovery suites and comparison artifact production) and `pages.yml` (verified artifact download, site checks/build and Pages deployment).

.github/workflows/Features/BenchmarkComparisons/QualifySite/AGENTS.md

@@ -33,3 +33,7 @@
 - The current four workflows are Build and Tests (`build-and-tests.yml`), Benchmarks (`benchmarks.yml`), Website (`website.yml`) and prepared manual Release (`release.yml`). Only Website builds, qualifies and publishes the site on trusted main push/manual; Benchmarks ends with a bounded Website dispatch and Website has no `workflow_run` executor subscription. These explicit owner corrections supersede every earlier three-workflow, filename, executor, placement and completion-subscription clause; every unrelated suite, permission and qualification rule remains mandatory.
 - The sole measured producer contract is the current 1,386-worker/2,530-input cohort in ADR-076. Its executed dependency manifest contains exactly 80 current source paths. Remove old-plan readers, schemas, event-file parsers and their exclusive fixtures or inventory entries under the root owner-only migration/legacy super rule. This supersedes earlier 270/277 and old-reader requirements only; retain bounded authenticated REST producer/artifact provenance, exact current source and input hashes, failed/null accounting, strict selected-evidence rejection, freshness and unchanged 80/70/90 coverage thresholds.
 - When no current authenticated producer is ready, qualify the complete content-only site without figures. Real native TUnit/Node/Chrome operations, no skips in each applicable suite, source/coverage inventories and needs-gated least-privilege Pages remain required. Local tests enter the same Aspire-owned AppHost and are development evidence; genuine exact-source Linux/provider proof closes delivery. Controlled rejection inputs cannot become published measurements.
+
+## Test project separation, owner correction 2026-10-06
+
+- Website startup/optional-selection checks moved with the benchmark infrastructure contracts to KeyLoad.ComparisonTests. Build that project and select only those two gates through the Aspire comparison suite. Preserve their complete assertions, TRX and Website executor checks; this does not execute database benchmark workloads. Ordinary Build and Tests must not execute them.

.github/workflows/Features/BenchmarkComparisons/QualifySite/action.yml

@@ -123,9 +123,9 @@ runs:
         pwsh --version > "$EVIDENCE_DIR/powershell-version.txt"
         dotnet restore KeyLoad.slnx
         dotnet build src/KeyLoad.AppHost/KeyLoad.AppHost.csproj --no-restore --configuration Release
-        dotnet build tests/KeyLoad.UnitTests/KeyLoad.UnitTests.csproj --no-restore --configuration Release
-        dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/SiteQualificationStartupTests/*' --KeyLoadTests:ReportTrx=true "--KeyLoadTests:ResultsDirectory=$EVIDENCE_DIR/startup-tests"
-        dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/SiteOptionalBenchmarkSelectionTests/*' --KeyLoadTests:ReportTrx=true "--KeyLoadTests:ResultsDirectory=$EVIDENCE_DIR/selection-tests"
+        dotnet build tests/KeyLoad.ComparisonTests/KeyLoad.ComparisonTests.csproj --no-restore --configuration Release
+        dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/SiteQualificationStartupTests/*' --KeyLoadTests:ReportTrx=true "--KeyLoadTests:ResultsDirectory=$EVIDENCE_DIR/startup-tests"
+        dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/SiteOptionalBenchmarkSelectionTests/*' --KeyLoadTests:ReportTrx=true "--KeyLoadTests:ResultsDirectory=$EVIDENCE_DIR/selection-tests"
         dotnet build tests/KeyLoad.Analyzers.Tests/KeyLoad.Analyzers.Tests.csproj --no-restore --configuration Release
         mkdir -p "$EVIDENCE_DIR/analyzer-coverage"
         cp scripts/Features/CodeQuality/site-analyzer-coverage.settings.xml "$EVIDENCE_DIR/analyzer-coverage/coverage.config.xml"

.github/workflows/benchmarks.yml

@@ -11,7 +11,7 @@ jobs:
   comparison-build:
     name: Build and check KeyLoad
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 120
     steps:
       - name: Download source code
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -27,6 +27,21 @@ jobs:
         run: dotnet format KeyLoad.slnx --verify-no-changes --no-restore
       - name: Check repository rules
         run: node scripts/Features/RepositoryGovernance/verify.mjs
+      - name: Test benchmark contracts
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/KeyLoad.UnitTests*/*/*' --KeyLoadTests:ResultsDirectory=TestResults/benchmark-contracts --KeyLoadTests:ReportTrx=true
+      - name: Test benchmark contracts without CPU intrinsics
+        env:
+          DOTNET_EnableHWIntrinsic: '0'
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/KeyLoad.UnitTests*/*/*' --KeyLoadTests:ResultsDirectory=TestResults/benchmark-contracts-scalar --KeyLoadTests:ReportTrx=true
+      - name: Test prompt termination on Aspire resource failures
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/AspireFailure*/*' --KeyLoadTests:ResultsDirectory=TestResults/aspire-failure --KeyLoadTests:ReportTrx=true
+      - name: Save benchmark contract test results
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
+        with:
+          name: benchmark-contract-tests
+          path: TestResults/**
+          if-no-files-found: error
       - name: Save build diagnostics
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -84,10 +99,8 @@ jobs:
       - name: Build benchmark tests
         id: benchmark-build
         run: dotnet build tests/KeyLoad.ComparisonTests --no-restore --configuration Release
-      - name: Build benchmark preparation unit tests
-        run: dotnet build tests/KeyLoad.UnitTests --no-restore --configuration Release
       - name: Check bounded Docker registry readiness
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ImageRegistryReadinessTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/registry-readiness
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ImageRegistryReadinessTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/registry-readiness
       - name: Build KeyLoad server and benchmark Docker images
         id: images
         run: node scripts/Features/BenchmarkComparisons/prepare-images.mjs
@@ -141,9 +154,9 @@ jobs:
       - name: Test live benchmark progress capture
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-progress
       - name: Test live benchmark process entry
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/NativeBenchmarkProgressEntryTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-entry
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/NativeBenchmarkProgressEntryTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-entry
       - name: Test benchmark phase and attempt counters
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ComparisonLiveProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-counters
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonLiveProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-counters
       - name: Test KeyLoad replay admission
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedKeyLoadReplayAdmissionTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/replay-admission
       - name: Test Redis replica error reporting

.github/workflows/build-and-tests.yml

@@ -144,14 +144,6 @@ jobs:
         run: node scripts/Features/RepositoryGovernance/verify.mjs
       - name: Test code analyzers
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=analyzers --KeyLoadTests:ReportTrx=true
-      - name: Test prompt termination on Aspire resource failures
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/AspireFailure*/*' --KeyLoadTests:ResultsDirectory=TestResults/aspire-failure --KeyLoadTests:ReportTrx=true
-      - name: Test live benchmark progress capture
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/live-benchmark-progress --KeyLoadTests:ReportTrx=true
-      - name: Test live benchmark process entry
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/NativeBenchmarkProgressEntryTests/*' --KeyLoadTests:ResultsDirectory=TestResults/live-benchmark-entry --KeyLoadTests:ReportTrx=true
-      - name: Test benchmark phase and attempt counters
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ComparisonLiveProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/live-benchmark-counters --KeyLoadTests:ReportTrx=true
       - name: Test native coverage source ownership
         id: source-ownership
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/(NativePathMapCompilerTests)|(ProductionSourceManifestSettlementTests)|(NativeSourceManifestChildSettlementTests)|(ProductionSourceManifestOperationTests)/*' --KeyLoadTests:ResultsDirectory=TestResults/native-source-ownership --KeyLoadTests:ReportTrx=true
@@ -212,17 +204,16 @@ jobs:
           global-json-file: global.json
       - name: Check Docker
         run: docker version
-      - name: Build KeyLoad server and benchmark Docker images
+      - name: Build KeyLoad server Docker image
         id: images
-        run: node scripts/Features/BenchmarkComparisons/prepare-images.mjs
+        run: node scripts/Features/BenchmarkComparisons/prepare-images.mjs --server-only
       - name: Configure Docker image references
         shell: bash
         env:
           KEYLOAD_SERVER_IMAGE: ${{ steps.images.outputs.server-image }}
-          KEYLOAD_RUNNER_IMAGE: ${{ steps.images.outputs.load-generator-image }}
         run: |
-          test -n "$KEYLOAD_SERVER_IMAGE" && test -n "$KEYLOAD_RUNNER_IMAGE"
-          printf 'KeyLoad__ContainerImages__Server=%s\nBenchmarks__ContainerImages__LoadGenerator=%s\nKEYLOAD_IMAGE_RECEIPT=%s\n' "$KEYLOAD_SERVER_IMAGE" "$KEYLOAD_RUNNER_IMAGE" "$RUNNER_TEMP/keyload-images/image-receipt.json" >> "$GITHUB_ENV"
+          test -n "$KEYLOAD_SERVER_IMAGE"
+          printf 'KeyLoad__ContainerImages__Server=%s\nKEYLOAD_IMAGE_RECEIPT=%s\n' "$KEYLOAD_SERVER_IMAGE" "$RUNNER_TEMP/keyload-images/image-receipt.json" >> "$GITHUB_ENV"
       - name: Find Chrome for admin tests
         shell: bash
         run: |

AGENTS.md

@@ -537,3 +537,7 @@ A bounded website qualification candidate contains the20-project historical runt
 ## Build and Tests workflow filename, owner direction 2026-10-06
 
 - The canonical Build and Tests workflow is `.github/workflows/build-and-tests.yml`, renamed from `ci.yml`. Dispatch with `gh workflow run build-and-tests.yml --repo managedcode/KeyLoad --ref main`. This supersedes earlier active filename/dispatch references only; preserve all build/test gates, stable job IDs, permissions and historical evidence.
+
+## Build and Tests scope, owner correction 2026-10-06
+
+- Build and Tests MUST execute only solution builds, repository checks and KeyLoad functional tests, including analyzer, unit/scalar, process recovery and Aspire RF3 SDK/MCP tests. Benchmark/comparison tests, progress/entry/counter checks, load generators and comparison-image preparation MUST NOT execute there, even when stored in the unit-test project. Run them exclusively in the separate Benchmarks workflow; preserve their checks there. This explicit correction supersedes earlier placement of benchmark correctness checks in ordinary CI. Compiling the complete solution remains mandatory.

benchmarks/KeyLoad.BenchmarkScenarios/KeyLoad.BenchmarkScenarios.csproj

@@ -10,6 +10,6 @@
     <PackageReference Include="BenchmarkDotNet" />
     <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" />
     <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
-    <InternalsVisibleTo Include="KeyLoad.UnitTests" />
+    <InternalsVisibleTo Include="KeyLoad.ComparisonTests" />
   </ItemGroup>
 </Project>

benchmarks/KeyLoad.ComparisonHost/Features/BenchmarkComparisons/Hosting/ComparisonHostTestFriend.cs

@@ -6,5 +6,5 @@ namespace KeyLoad.ComparisonHost.Features.BenchmarkComparisons;
 
 internal static class ComparisonHostTestFriend
 {
-    internal const string AssemblyName = "KeyLoad.UnitTests";
+    internal const string AssemblyName = "KeyLoad.ComparisonTests";
 }

docs/ADR/ADR-074-aspire-owned-test-entry.md

@@ -16,6 +16,22 @@ minutes after original run37349838022 cancelled recovery at the old job limit.
 Keep unit, scalar and recovery sequential, every suite's native deadline and all
 required source/runtime/artifact gates. No timeout or failed suite becomes green.
 
+## Build and Tests boundary, owner correction 2026-10-06
+
+REQ/AC-TEST-PIPELINE-001 requires only functional KeyLoad suites in Build and
+Tests. Benchmark contract cases and exclusive helpers move from UnitTests to
+ComparisonTests/Features/BenchmarkComparisons/UnitContracts, preserving
+namespaces/assertions. Benchmarks owns their normal/scalar Aspire comparison
+execution and original TRX artifacts. Shared pure infrastructure retains one
+source compiled by both executables. Root owns TASK-TEST-PIPELINE-001..003 and
+integration. RF3 selects `prepare-images.mjs --server-only`: build/push/hash only
+the server image, with the same source-bound receipt/manifest and scoped registry
+cleanup. Benchmarks keeps both server and load-generator preparation. No database
+API or server-record receipt format changes. Verify native image-mode selection
+and rejection, transferred discovery inventory, solution build, formatter,
+governance and Aspire functional suites. Failed gates remain open; reverting
+this boundary requires owner direction.
+
 ## Decision and boundaries
 
 `KeyLoad.AppHost --KeyLoadTests:Suite=<suite>` composes one actual Aspire

docs/ADR/ADR-085-live-benchmark-progress.md

@@ -31,8 +31,8 @@ timeout would hide slow execution and cannot qualify the workload.
 4. Root owns the process relay scripts, workflow/test integration and docs. Retain
    canonical dotnet/AppHost invocation, original exit and signal handling. No
    independent database startup or global process termination is permitted.
-   CI runs focused native progress capture regressions independently of the
-   complete Benchmarks matrix; these do not qualify a database workload.
+   Benchmarks runs focused native progress capture regressions independently of
+   its complete measurement matrix; these do not qualify a database workload.
 5. Root reviews every diff, builds, runs real focused Aspire/TUnit checks, format
    and governance; commits only this repair and pushes under standing authority.
    Actual Linux CI/native workload/cancellation artifacts supply delivered proof.