Skip to content

test(executor): deflake TestQueueTask_ConcurrentDuplicate_DispatchesOnce (GH-4513)#4516

Merged
alekspetrov merged 1 commit into
mainfrom
pilot/GH-4513
Jul 23, 2026
Merged

test(executor): deflake TestQueueTask_ConcurrentDuplicate_DispatchesOnce (GH-4513)#4516
alekspetrov merged 1 commit into
mainfrom
pilot/GH-4513

Conversation

@alekspetrov

Copy link
Copy Markdown
Collaborator

Summary

Fixes GH-4513: TestQueueTask_ConcurrentDuplicate_DispatchesOnce/canary-sandbox-style_path (internal/executor/dispatcher_test.go) was flaky and today killed a correct PR (#4509 / GH-4507) via the ci_pre_merge auto-close path.

Root cause

The test wired the dispatcher to NewRunner() — the real ClaudeCodeBackend — instead of a mock. dispatchMu only serializes QueueTask's duplicate check + insert; it does not cover what happens after. The winning QueueTask call's ensureWorker() starts a background ProjectWorker goroutine that immediately picks up the freshly-queued row, transitions it to "running", and calls runner.Execute against the test's non-existent project path (/Users/pilot-op/Projects/startups/pilot...). Against the real backend, Execute's preflight check (git status chdir) fails fast — observed ~0.5-1s locally, confirmed by direct probe — and marks the row terminal ("failed").

IsTaskQueued only treats queued/pending/running as "active". If that terminal transition landed before all 8 concurrent goroutines made it through dispatchMu's serialized check, a "loser" goroutine checking afterward no longer saw the row as active and fell through to beginWithGenerationRetry's legitimate dead-claim repick path — minting a second real dispatch and a second nil-error QueueTask return, inflating successes past 1. Since dispatchMu fully serializes the 8 calls' fast SQL work but the real backend's failure latency isn't bounded relative to that, this is a genuine race whose outcome depends on CI scheduler/load — exactly the kind of thing that doesn't reproduce reliably locally (confirmed: 100 local runs all passed before the fix) but bites under CI conditions.

This is unrelated to PR #4509/GH-4507's actual change (internal/executor/intent_judge.go, preflight-cap logic) — the subtest failure was pure test-harness flake, not a regression from that PR.

Fix

Swap in a blockingBackend (Execute blocks on <-ctx.Done()) with skipPreflightChecks/config set the same way sibling tests in this file already do for mock backends. This pins the winning execution row at "running" for the entire concurrent-dispatch assertion window deterministically — no sleeps, no timing assumptions — removing the real backend's uncontrolled completion latency as a race variable entirely. The row only transitions once dispatcher.Stop() cancels the context, which happens after all assertions in both subtests have already completed.

No assertion was weakened — both subtests still verify exactly one dispatch out of concurrency (8) concurrent QueueTask calls.

Test plan

  • go build ./...
  • go test ./internal/executor/... (full package, includes new-etc)
  • go test -race -run 'TestQueueTask_ConcurrentDuplicate_DispatchesOnce' -count=200 ./internal/executor/ — passes
  • go test -race -run 'TestQueueTask_ConcurrentDuplicate_DispatchesOnce' -count=50 -cpu 1,2,4 ./internal/executor/ — passes
  • gofmt -l clean on the changed file

…nce (GH-4513)

Root cause: the test wired the dispatcher to NewRunner()'s real
ClaudeCodeBackend. The winning QueueTask call's ensureWorker() starts a
background ProjectWorker goroutine outside dispatchMu's protection that
immediately runs the real backend against the test's non-existent project
path; its preflight check fails fast (observed ~0.5-1s locally) and marks
the execution row terminal. If that terminal transition landed before all
8 concurrent goroutines made it through dispatchMu's serialized
IsTaskQueued check, a "loser" goroutine no longer saw the row as
queued/running and fell through to the legitimate dead-claim repick path,
minting a second real dispatch and inflating `successes` past 1 — exactly
what killed PR #4509/GH-4507 despite that change never touching dispatch
concurrency.

Fix: swap in a blockingBackend that never returns from Execute until the
test tears the dispatcher down (context cancellation), with
skipPreflightChecks/config set the same way sibling tests in this file
already do for mock backends. This pins the row at "running" for the
entire assertion window deterministically, removing the real backend's
uncontrolled completion latency as a timing dependency. No assertion was
weakened; -count=200 -race and -cpu 1,2,4 both pass.
@alekspetrov
alekspetrov merged commit c3258c3 into main Jul 23, 2026
5 checks passed
@alekspetrov
alekspetrov deleted the pilot/GH-4513 branch July 23, 2026 15:38
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(dispatcher): deflake TestQueueTask_ConcurrentDuplicate_DispatchesOnce/canary-sandbox-style_path — killed correct PR #4509

2 participants