Tile PG-target TPS, sessions, CPU and Aurora waits detectors (#3653 A8 option B, lane L3a)

[erikdarlingdata]

Part of #3653 (A8 option B, lane L3a).

What this PR does

Moves the four PG-target families this lane owns onto per-hour tiles, scored through AnomalyGate.EvaluateTiles against PgTargetBaselineProvider.GetBucketMapAsync, with the never-blind fallback to today's whole-window EvaluateZScore when no tile clears AnomalyThresholds.MinTileSamples or has a non-empty bucket:

• TPS (DetectTpsAnomalies): new TpsTileWindowSql const. DatabaseCounterWindowSql is left byte-identical — the deadlock-rate detector still reads it, unchanged.
• Sessions (DetectSessionAnomalies): new SessionTileWindowSql, keeping the per_collection CTE and grouping the outer SELECT by tile.
• CPU (DetectCpuAnomalies): new CpuTileWindowSql. Peak time moves to a per-tile array_agg(collection_time ORDER BY v DESC)[1]. The raw cpu_percent window scalar rides per tile; the detector reports the worst tile's raw value.
• Aurora waits (DetectWaitProfileAnomalies): new WaitRateTileWindowSql. Rewritten in this lane to follow the coordinator's ruling on multi-arm families (see below), replacing L3a's original per-tile three-arm decision.

L3a-2 (this lane): work done

Step 1 — Aurora waits conforms to the coordinator's ruling

L3a's original code ran all three trust arms (robust z / classical ratio / absolute fallback) per tile by hand through a private DecideWaitProfileTile/WaitProfileTileDecision, then picked the worst firing tile. That broke the design in the four ways the brief named: no multiplicity correction on the ratio arm, an uncorrected k on the z arm's per-tile mean, a single untrustworthy tile able to switch the whole window to the absolute arm even when the start bucket trusted the z arm, and duplicated gate logic.

Replaced with the ruling's shape:

• The arm is chosen from the START-hour bucket (GetBaselineAsync at TimeRangeStart), exactly as dev's whole-window body decides it.
• Only arm 1 (baseline.IsTrustworthy && EffectiveRobustSigma > 0) moves onto tiles: AnomalyGate.EvaluateTiles(tiles, map, HeavyTailModifiedZThreshold, HeavyTailModifiedZThreshold, PgWaitProfileFallbackMsPerSec, PgWaitProfileFallbackMsPerSec, SigmaDisplayCap, window). When it returns null, today's EvaluateZScore runs over WindowTiles.WholeWindow(tiles).
• Arms 2 and 3 (classical ratio, absolute fallback) stay whole-window and unchanged, fed from WholeWindow(tiles)'s Peak/Mean — no second SQL read. Commented in the source as outside B's scope.
• DecideWaitProfileTile/WaitProfileTileDecision deleted.
• Metadata (modified_z, mean_modified_z, ratio, fire_threshold, baseline keys) comes from the bucket and values actually scored — the worst tile's on the tiled path, the start bucket's otherwise. AddTileMetadata runs only on the tiled path.
• Builds clean (dotnet build Darling/PerformanceMonitor.Darling.Analysis -p:EnableWindowsTargeting=true → 0 errors).

Step 2 — pins and census

• PgTargetAnomalyTests.s_detectorSql: added TpsTileWindowSql, SessionTileWindowSql, CpuTileWindowSql, WaitRateTileWindowSql (the reflection census requires every public const ending Sql to be listed).
• Added table/local_hour/dialect pins for each of the four new consts (same file, EveryDetectorRead_... test).
• Found and fixed a pre-existing pin regression from L3a's earlier commits (not introduced by this lane's step 1, but in scope for "fix every pin failure" — step 2's own instruction): ThePairGate_... test's regex pinned AnomalyGate.EvaluateZScore(baseline, peakTps, avgTps, ... window: context.TimeRangeEnd - context.TimeRangeStart) verbatim, but L3a's TPS/sessions/CPU tiled call-flow refactor introduced a local window variable computed once at the top of each method (also feeding EvaluateTiles), so the call site now reads window: window. Updated the three regexes (TPS/sessions/CPU) to match window: window and added a pin that the local is assigned context.TimeRangeEnd - context.TimeRangeStart. Applied the same fix to the wait-profile detector's own EvaluateZScore pin after step 1's rewrite.
• LocalClockBucketKeyTests's "$4..$6" census does NOT cover these consts — checked: that census (EveryBucketStatement()) only enumerates PgBaselineProvider/PgTargetBaselineProvider baseline queries, not detector window-SQL consts. Confirmed against lane L3b-2's own step-1 commit (1e9c8ce76), which touched pins for its own tiled families in this same PR wave and made no change to that census either. No action needed there.
• Grepped Darling/Darling.Tests for the old const names and for DetectTpsAnomalies/DetectSessionAnomalies/DetectCpuAnomalies/DetectWaitProfileAnomalies text pins and window_samples — no other pin needed updating.
• Verified: dotnet Darling.Tests.dll -class Darling.Tests.PgTargetAnomalyTests → 44 total, 0 failed, 2 skipped (live-PG classes, expected without DARLING_TEST_PG). -class LocalClockBucketKeyTests -class PgTargetClockTests -class WindowTilesTests -class AnomalyGateTilesTests → 41 total, 0 failed.

What is NOT done (ran out of context before steps 3 and 4)

3. Live PG run — not run. Needs timescale/timescaledb:2.28.1-pg18 on a free 558xx port, CREATE ROLE darling ..., DARLING_TEST_PG set, then *PgTargetAnomaly*, *PgTargetBetweenWaves*, *PgTargetWaitProfile*, *PgTargetClock* run against it (strip Microsoft.WindowsDesktop.App from the runtimeconfig first, restore after). This is the recipe's own "Skipped: N is not a pass" requirement — the tiled SQL for all four families has never executed against a real PostgreSQL. Highest-priority remaining item.
4. Timing — not run. Seed 24h of one-per-minute session rows, time SessionTileWindowSql vs. dev's SessionWindowSql (5 runs each, median), both numbers in this PR body.
5. Full Darling suite once with DARLING_TEST_PG unset — not run in this lane's pass (ran only the targeted classes above). dev has 214 Windows-only failures on this Mac per the brief; any failure outside that set needs naming.
6. Lite build sanity check — not done (out of scope per L3a's own note, this lane didn't touch it either).
7. New behaviour tests (planted 2h +6σ shift, tile_local_hour assertions) — explicitly out of scope per the brief ("a follow-up lane writes them") unless steps 1-4 finished with room to spare; they did not.

Handoff

The next lane/coordinator pass should, in order: (a) run step 3's live-PG proof — this is the acceptance-blocking item, since the tiled Aurora-waits rewrite in step 1 has never run against real PostgreSQL and the ruling's arm-choice logic (start-bucket decides, only arm 1 tiles) is exactly the kind of thing that looks right by inspection and needs a live check; (b) run step 4's timing and drop both numbers in here; (c) then the full suite once.

CHANGELOG entry

• The PostgreSQL-target TPS, session-count, CPU-capacity and Aurora all-types wait-profile anomaly detectors score each hour of a window against its own hour-of-week baseline, falling back to the whole-window comparison when no hour has enough samples; the wait-profile family's classical-ratio and no-baseline arms stay whole-window per the coordinator's multi-arm ruling (Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653).

Base branch note

Built on origin/feat/3653-b-bucket-map (#4168's head) because GetBucketMapAsync was not yet on origin/dev at the time this lane started. Per the lane brief, titled DO NOT MERGE (stacked on #4168); the coordinator rebases onto dev once #4168 lands. (#4168 has since merged to dev as 41a10c729 — a rebase may now be straightforward, but this lane did not attempt it, out of scope for L3a-2's brief.)

Behaviour tests

Darling/Darling.Tests/PgTargetTileBehaviourTests.cs: all 7 pass live (timescale/timescaledb:2.28.1-pg18).

• Sessions_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuiet — PASS
• Sessions_TwoHourShiftInTwentyFourHourWindow_Fires_WithTheRaisedCutoff — PASS
• Sessions_LoneSpikeInFourHourWindow_DoesNotFire — PASS
• Sessions_SparseWindow_FallsBackToWholeWindowPath — PASS
• Tps_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuiet — PASS (fixed this lane)
• Cpu_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuiet — PASS
• WaitProfile_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuiet — PASS

TPS root cause (diagnosed, then fixed): the fixture's minute-resolution rows only ran across minutes 0–11
of each hour, leaving a ~49-minute gap to the next hour's first sample. The per-collection LAG-rated tps for
that boundary-crossing sample divides one minute-increment's worth of xacts by the ~49-minute gap, so it comes
back near 1–2 tps instead of ~60–76 — a spurious low outlier inside every tile. That single low sample dragged
each tile's AVG below the correctMean-raised cutoff even though its PEAK cleared easily, so no tile's
Decision.Fire was true (EvaluateTiles still returned a non-null, non-firing verdict — the "still returns a
display verdict even on a quiet pass" contract, not the never-blind-null path), and DetectTpsAnomalies
returned with no fact. Diagnosed with a temporary [Fact] (never committed) that ran TpsTileWindowSql's own
6 binds directly and printed the raw per-collection rated rows, the bucket map per window hour, and the fired
fact count.

Fix: the TPS fixture's rows are now spaced every 5 minutes across the full hour (i in 0..11, matching the
sessions/CPU fixtures' own convention) instead of every 1 minute, with the per-step increment scaled from *60
to *300 to keep the seeded μ/σ tps unchanged under the new 300-second step. Two INSERT statements changed
(PlantTpsHistoryAsync, PlantTpsWindowAsync); no product code touched.

Mutation check: AnomalyGate.EvaluateTiles forced to always return null → the 5 tile-shift-dependent
tests (sessions 4h, sessions 24h, CPU, TPS, wait profile) all failed as expected; the 2 non-tile-dependent
tests (Sessions_LoneSpikeInFourHourWindow_DoesNotFire, Sessions_SparseWindow_FallsBackToWholeWindowPath)
stayed green. Reverted; rebuilt; git diff --stat PerformanceMonitor.Analysis/AnomalyGate.cs is empty.

Full suite (DARLING_TEST_PG unset): 13667 total, 214 failed, 737 skipped — matching dev's known
214-Windows-only-failure baseline on this Mac exactly. *PgTargetAnomaly* + *PgTargetTileBehaviour* live:
51/51 pass.

Not verified: CI (Windows) run of these tests — only this Mac's live-PG rig and the local suite.

[erikdarlingdata]

Behaviour tests (lane T4171-2, tests-only)

Landed T4171's PgTargetTileBehaviourTests.cs (cherry-picked from 8a87e96d7) onto this branch with a plain push, then made it build and iterated it live against timescale/timescaledb:2.28.1-pg18.

Result: 6 of 7 pass live. 1 (TPS scenario 1) is still red — pushed as-is per the brief's "push scenario 1, iterate" rule, but I ran out of budget before root-causing it. Flagging for a fast follow-up, not closing the loop myself.

What I found and fixed

1. The as-written test asserted the wrong "dilution" baseline. Every scenario-1 test computed wholeWindowPeakZ from the RAW SHIFT VALUE against the start bucket, not from the window's own MEAN (baseline hours + shifted hours). Judging the peak alone is trivially anomalous by construction — it doesn't prove dilution. Fixed all three (Sessions_TwoHourShiftInFourHourWindow, Tps_TwoHourShiftInFourHourWindow, Cpu_TwoHourShiftInFourHourWindow) to compute the actual per-collection window mean (matching the seeded sin-noise shape, rounded where the product rounds) and assert THAT clears the cutoff — the real "two quiet hours dilute two shifted hours" case.
2. Missing canary gate seed. PgTargetAnomalyDetector.DetectAnomaliesAsync gates on HasBaselineDataAsync, which reads pg_database_stats ALONE (30-day lookback) before ANY detector runs — sessions and CPU tests never seeded that table, so the whole detector returned empty facts regardless of what was in pg_session_states/pg_cpu_utilization. Added PlantCanaryDatabaseStatsRowAsync and wired it into the 4 sessions scenarios and the CPU scenario (TPS and wait-profile already write to pg_database_stats directly, so they didn't need it).
3. CPU shift magnitude was too large. 90% shift against a 30±2% baseline gave a whole-window mean z of 6.0 (past the 3.5 cutoff) — not diluted by construction. Reduced the shift to 60% (still clears PgCpuFloorPct at 40).

Still red: TPS scenario 1

After fixing the canary gate and re-sizing the shift (76 tps vs 60±3 baseline, chosen so the computed window mean sits below the 3.5σ cutoff against the measured bucket), the detector still returns an EMPTY fact list — Assert.Single(facts, f => f.Key == PgTargetFactKeys.AnomalyTps) fails on an empty collection, same symptom as the missing-canary bug but the canary is present here (TPS seeds pg_database_stats directly via PlantTpsHistoryAsync/PlantTpsWindowAsync). By hand-computing the tile bucket stats (median ≈ 60, robust sigma ≈ 3.1-3.2) the hour-3/hour-4 tiles' peak and mean sigma both come out around 5.0–5.2 — comfortably past the 3.5 cutoff, so it SHOULD fire. Not yet root-caused: could be the LAG-based TpsTileWindowSql's per-database windowing losing a row at an hour boundary, MinTileSamples (3) not being cleared for one tile, or a units/interval mismatch I haven't isolated. This is a test-precondition problem until proven otherwise (per the brief, a live failure after preconditions pass would be a product bug — I have not yet confirmed preconditions pass here).

Live verification

• timescale/timescaledb:2.28.1-pg18 on port 55890, role darling created, DARLING_TEST_PG set.
• Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj -p:EnableWindowsTargeting=true — 0 errors both times.
• Microsoft.WindowsDesktop.App stripped from Darling.Tests.runtimeconfig.json after every build (never committed).
• dotnet Darling.Tests.dll -class 'Darling.Tests.PgTargetTileBehaviourTests' (final targeted run): Total: 7, Errors: 0, Failed: 1, Skipped: 0 — the 6 passing: Sessions_TwoHourShiftInFourHourWindow_FiresPerTile…, Sessions_TwoHourShiftInTwentyFourHourWindow_Fires…, Sessions_LoneSpikeInsideAnHour_DoesNotFire, Sessions_UnderMinTileSamplesEveryTile_FallsBackToTheWholeWindowPath…, Cpu_TwoHourShiftInFourHourWindow_FiresPerTile…, WaitProfile_TwoHourShiftInFourHourWindow_FiresPerTile…. Failing: Tps_TwoHourShiftInFourHourWindow_FiresPerTile….
• Not done (budget ran out before turn 8's follow-through): the mutation check (AnomalyGate.EvaluateTiles → null) and the full *PgTargetAnomaly* live class, plus the full Darling suite with DARLING_TEST_PG unset. Container was removed at end of session.
• Did NOT touch product code — only the test file, per the tests-only brief.

CHANGELOG entry

(none — tests-only lane; no user-facing change)

Handoff

Next lane should: (1) root-cause the TPS scenario-1 failure (instrument TpsTileWindowSql's tile output directly, e.g. add a temporary diagnostic query against the live rig, before touching test values further); (2) run the mutation check and the two full-suite passes the brief calls for; (3) if TPS turns out to be a genuine product bug, push it as a RED: … commit per the scenario brief instead of tuning the test further.