release-26.3: pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed

[cockroach-teamcity]

pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestExecBuild_distsql_plan_locality_filter failed on release-26.3 @ e83d5ed5cc10a4601448eec2a6810ed98de6cab5:

=== RUN   TestExecBuild_distsql_plan_locality_filter
    test_log_scope.go:172: test logs captured to: outputs.zip/logTestExecBuild_distsql_plan_locality_filter2011314841
    test_log_scope.go:82: use -show-logs to present logs inline

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-65105

[mw5h]

/investigate

[github-actions]

Investigation: TestExecBuild_distsql_plan_locality_filter

Investigated failure: issue body
Failure SHA: e83d5ed5cc10a4601448eec2a6810ed98de6cab5
Confidence: high

Warning: Could not check out failure SHA e83d5ed5cc10a4601448eec2a6810ed98de6cab5. Analysis is based on the default branch tip. The relevant test file and testdata are unchanged between the failure branch and tip.

What This Test Does

This test exercises the distsql_plan_locality_filter session variable in a 9-node, 3-region in-process cluster. It creates a table, splits it into 3 ranges across regions, and verifies that EXPLAIN (VEC) plans only schedule DistSQL flows on nodes matching the configured locality filter.

Where the Failure Occurs

The test times out after 1h16m40s during cluster setup in newCluster at logic.go:1945. The test goroutine is blocked waiting for a SQL response to SET CLUSTER SETTING sql.stats.automatic_collection.min_stale_rows, stuck in pq.(*conn).readBindResponse. The server never responds because the 9-node cluster is severely overloaded.

The EngFlow invocation ran 3 attempts:

• Run 1: Killed after ~31 minutes (exit code -1), no test output beyond initial setup.
• Run 2: Timed out after 1h16m40s with full goroutine dump confirming the blocked state.
• Run 3: Also killed (exit code -1).

Analysis

Hypothesis: Race-detector overhead causes timeout on resource-constrained CI (likely cause, high confidence)

The test runs with race=true and spins up a 9-node in-process cluster. The race detector imposes extreme CPU and memory overhead. The cluster never stabilizes enough to complete even the setup phase. Evidence from the run 2 server logs:

• 87 under-replicated ranges persisting throughout: ranges.underreplicated@s1=87
• 18-22 second latch holds: has held latch for 18.2s, 22.1s, 21.2s
• Slow heartbeats >1s: slow heartbeat took 1.053s, 1.465s, 1.299s
• "node might be overloaded" warnings from raft ready handling
• ~1,982 overload-related log messages in the server log

The test has skip.UnderDeadlock(t, "times out and/or hangs") at generated_test.go:58 but does not have skip.UnderRace. Adding skip.UnderRace(t, "9-node cluster times out under race detector") would prevent this failure.

This test validates query plan correctness, not race conditions. Running a 9-node multi-region in-process cluster under the race detector provides minimal incremental value over the non-race run.

Recommendation: Add skip.UnderRace(t, "9-node cluster times out under race detector") to the runExecBuildLogicTest function in generated_test.go:58 (or equivalently to the test generation template). This matches the existing skip.UnderDeadlock pattern. No fix PR has been filed yet despite multiple prior investigations recommending this.

Related Issues and PRs

• #171910 (closed, release-26.3) — Same test, same timeout-under-race failure mode, investigated just yesterday with identical findings.
• #171633 (open, master) — Same test, same failure mode. Has a prior investigation with identical findings. 8+ failures since June 13, all race=true.
• #171782 (closed, release-26.3) — Same test on release-26.3, closed as dupe of pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed #171633. Has another investigation with the same conclusion. 5+ failures.
• #170347 (closed) — Original test assertion flakiness (plan output mismatch), fixed by 68071d846b0 / #170401. Different failure mode — the current issue is a timeout, not an assertion mismatch.
• #170414 (open, P-3, master) — Initially filed for a DATA RACE in SetYield, but evolved into the same overloaded-cluster timeout pattern. The data race was separately fixed.
• #170847 (closed) — SetYield data race fix.

Timeline

Date	Event
2026-05-04	Test introduced on master
2026-05-15	Deflake fix merged (68071d846b0) — fixes assertion flakiness
2026-05-15	Deflake fix backported to release-26.3
2026-05-18 → 2026-06-04	DATA RACE in SetYield reported and fixed
2026-06-13 → present	Repeated race-build timeout failures on master (#171633) and release-26.3 (#171782, #171910, #171970)
2026-06-24	This issue filed: same failure mode on release-26.3

Detailed evidence and investigation notes

Log Excerpts

Timeout panic (test.log run 2, lines 12-15):

-- Test timed out at 2026-06-24 14:19:39 Universal --
panic: test timed out after 1h16m40s
	running tests:
		TestExecBuild_distsql_plan_locality_filter (1h16m40s)

Blocked test goroutine (test.log run 2, lines 102-145):

github.com/lib/pq.(*conn).readBindResponse(0xc03ebf1508)
	external/com_github_lib_pq/conn.go:1886 +0x5e
...
github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).newCluster(...)
	pkg/sql/logictest/logic.go:1945 +0x2e15

The blocked SQL statement is:

SET CLUSTER SETTING sql.stats.automatic_collection.min_stale_rows = 5

Server overload (run 2 server logs):

W260624 13:43:59.894318 ... has held latch for 18.2s ... node might be overloaded
W260624 13:44:02.140580 ... slow heartbeat took 1.053s
W260624 13:44:03.996501 ... slow heartbeat took 1.465s
W260624 13:44:05.785584 ... has held latch for 22.1s ... node might be overloaded
W260624 13:44:07.775435 ... health alerts detected: ranges.underreplicated@s1=87
W260624 13:44:10.917737 ... has held latch for 21.2s

Run 1 (EngFlow invocation 4e66eaeb, shard 1, run 1): Killed after ~31 minutes (exit code -1). No test output beyond initial setup.

Run 2 (shard 1, run 2): Timed out after 1h16m40s. Full goroutine dump captured. 1,982 overload-related messages in server log. 87 under-replicated ranges persisted throughout.

Run 3 (shard 1, run 3): Also killed (exit code -1).

Code References

• Test definition: generated_test.go:87-92
• Test runner with skip.UnderDeadlock but no skip.UnderRace: generated_test.go:56-67
• Test data file: distsql_plan_locality_filter
• Hang location: logic.go:1945

Things Ruled Out

• Test assertion flakiness: The original assertion bug (exact plan matching) was fixed in 68071d846b0. Current failures are timeouts, not assertion mismatches.
• DATA RACE in SetYield: This was a separate issue (#170414) that has been fixed. No WARNING: DATA RACE message appears in the logs.
• Code logic regression: The test file and testdata have not changed since the deflake fix. No recent changes could have introduced a regression.
• Infrastructure issue: The same failure pattern is consistent across many different SHAs, dates, and CI runs — it's reproducible whenever race=true.

Other Failure Occurrences

This is the same failure mode that has been recurring across 4+ open/closed issues:

• pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed #171633 (master): 8+ failures, all race=true, all timeouts
• release-26.3: pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed #171782 (release-26.3): 5+ failures, all race=true, all timeouts, closed as dupe
• release-26.3: pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed #171910 (release-26.3): Same as this issue, same failure mode, closed yesterday
• pkg/sql/opt/exec/execbuilder/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestExecBuild_distsql_plan_locality_filter failed [DATA RACE in SetYield] #170414 (master): Mixed DATA RACE + timeout failures, race was fixed, timeouts continue

All failures share: race=true, 9-node in-process cluster, overloaded server, timeout during setup or before completing test queries.

Tooling Feedback

• Could not check out failure SHA e83d5ed5cc10a4601448eec2a6810ed98de6cab5 (SHA not reachable from this remote — likely a release-26.3 branch commit not present in this clone's history). The test files were unchanged between the failure SHA's branch and the default branch tip, so this did not affect the investigation.

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.