Repository navigation
fix: guard the pause resume timers against concurrent access - #1243
Merged
Merged
Conversation
msafwankarim
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1242
The resume timers for
deployment.reloader.stakater.com/pause-periodlive in the package-level mapactiveTimers. Controller workers read and insert entries throughPauseDeploymentandCreateResumeTimer, and each resume timer deletes its entry from atime.AfterFuncgoroutine inResumeDeployment. Nothing synchronizes these, so a timer firing for one Deployment while a worker pauses another can stop the process withfatal error: concurrent map writes.This change adds a
sync.Mutexnext toactiveTimersand takes it around every map access:PauseDeployment: the existence check beforeHandleMissingTimer.CreateResumeTimer: the check and insert, held together so two callers can't both create a timer for the same Deployment.ResumeDeployment: the stop and delete.The lock is never held during a Kubernetes API call, and a timer callback that fires early only waits for
CreateResumeTimerto return.Testing
TestResumeTimersConcurrentAccessruns 8 goroutines that each create and remove a resume timer for their own Deployment 1000 times, then checks that no timers remain.Without the fix:
go test -count=1 ./internal/pkg/handler/ -run TestResumeTimersConcurrentAccessexited withfatal error: concurrent map writesorconcurrent map read and map writein 3 of 5 runs.go test -race -count=1 ./internal/pkg/handler/ -run TestResumeTimersConcurrentAccessfailed with 9DATA RACEwarnings.With the fix:
go test -race -count=1 ./internal/pkg/handler/ -run 'Pause|Resume|Timer'passed with no race warnings.go test -count=1 ./internal/pkg/handler/passed, andgo vet,gofmt -landgolangci-lint run ./internal/pkg/handler/...report no issues.The handler package needs a kubeconfig for its test setup. I ran the tests locally with a kubeconfig that points to an unreachable server, which is enough for these unit tests.
This only affects
master. Thev2branch handles pause expiry inDeploymentReconcilerand has no timer map.