Skip to content

fix: guard the pause resume timers against concurrent access - #1243

Merged
msafwankarim merged 1 commit into
stakater:masterfrom
RaphaelFakhri:fix/pause-timer-race
Oct 21, 2026
Merged

msafwankarim merged 1 commit into
stakater:masterfrom
RaphaelFakhri:fix/pause-timer-race

Conversation

@RaphaelFakhri

Copy link
Copy Markdown
Contributor

Fixes #1242

The resume timers for deployment.reloader.stakater.com/pause-period live in the package-level map activeTimers. Controller workers read and insert entries through PauseDeployment and CreateResumeTimer, and each resume timer deletes its entry from a time.AfterFunc goroutine in ResumeDeployment. Nothing synchronizes these, so a timer firing for one Deployment while a worker pauses another can stop the process with fatal error: concurrent map writes.

This change adds a sync.Mutex next to activeTimers and takes it around every map access:

  • PauseDeployment: the existence check before HandleMissingTimer.
  • CreateResumeTimer: the check and insert, held together so two callers can't both create a timer for the same Deployment.
  • ResumeDeployment: the stop and delete.

The lock is never held during a Kubernetes API call, and a timer callback that fires early only waits for CreateResumeTimer to return.

Testing

TestResumeTimersConcurrentAccess runs 8 goroutines that each create and remove a resume timer for their own Deployment 1000 times, then checks that no timers remain.

Without the fix:

  • go test -count=1 ./internal/pkg/handler/ -run TestResumeTimersConcurrentAccess exited with fatal error: concurrent map writes or concurrent map read and map write in 3 of 5 runs.
  • go test -race -count=1 ./internal/pkg/handler/ -run TestResumeTimersConcurrentAccess failed with 9 DATA RACE warnings.

With the fix:

  • The plain run passed 5 of 5 times.
  • go test -race -count=1 ./internal/pkg/handler/ -run 'Pause|Resume|Timer' passed with no race warnings.
  • go test -count=1 ./internal/pkg/handler/ passed, and go vet, gofmt -l and golangci-lint run ./internal/pkg/handler/... report no issues.

The handler package needs a kubeconfig for its test setup. I ran the tests locally with a kubeconfig that points to an unreachable server, which is enough for these unit tests.

This only affects master. The v2 branch handles pause expiry in DeploymentReconciler and has no timer map.

@msafwankarim
msafwankarim merged commit 7d30eaa into stakater:master Oct 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Reloader can crash with "concurrent map writes" when pause-period is used

2 participants