Skip to content

fix: remove the resume timer before resuming a paused deployment - #1220

Merged
SyedaFatimaKazmi merged 2 commits into
stakater:masterfrom
kaanisthatyou:fix/resume-timer-cleanup
Oct 2, 2026
Merged

SyedaFatimaKazmi merged 2 commits into
stakater:masterfrom
kaanisthatyou:fix/resume-timer-cleanup

Conversation

@kaanisthatyou

Copy link
Copy Markdown
Contributor

Fixes #1219

ResumeDeployment removed the timer entry from activeTimers only after the Get of the deployment succeeded. If that Get failed (for example the API server was briefly unavailable when the timer fired), the function returned early and left the already-fired timer in the map. PauseDeployment only calls HandleMissingTimer when there is no entry for the deployment, so the deployment stayed paused and was never retried.

This moves the timer removal to the start of ResumeDeployment. By the time it runs, the timer has either fired or doesn't exist (the HandleMissingTimer path). A failed attempt then leaves no entry behind, and the next change to the deployment goes through HandleMissingTimer again. This also covers the other early returns (IsPausedByReloader false, resume patch creation failing), which had the same problem.

Testing

TestResumeDeploymentRetriesAfterFailedGet uses a fake clientset with a reactor that fails get deployments once, and checks that:

  1. after the failed resume, the timer entry is gone;
  2. a following PauseDeployment call for the still-paused deployment (pause period expired) resumes it.

The handler package needs a cluster for its test setup, so I ran the suite through the repository's pull request workflow on my fork: https://github.com/kaanisthatyou/Reloader/actions/runs/34899925407. golangci-lint, make test (including --- PASS: TestResumeDeploymentRetriesAfterFailedGet), the E2E tests and the A/B load tests passed. Only "Build Docker UBI Image" failed, because the fork can't pull from its own ghcr.io namespace. Locally, go vet ./internal/pkg/handler/ passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EcwjgZmx8zbtn6PpwDY9A1

ResumeDeployment removed the timer entry only after it had fetched the
deployment. If that Get failed, the already-fired timer stayed in
activeTimers, so PauseDeployment never called HandleMissingTimer again
and the deployment stayed paused.

Fixes stakater#1219

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcwjgZmx8zbtn6PpwDY9A1

@SyedaFatimaKazmi SyedaFatimaKazmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. This is the right fix, and I like that it stays minimal.

I have left two minor comments below. None of them block the merge but they are small additions and a naming point. Happy to merge once you have had a look at them.

Comment thread internal/pkg/handler/pause_deployment.go
Comment thread internal/pkg/handler/pause_deployment_test.go Outdated
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Yyp6p2GWzkWKLsrxxJR1U
msafwankarim added a commit that referenced this pull request Sep 17, 2026
Guards the v2 equivalent of the recovery PR #1220 restores on master. v2 keeps
no timer map, so a failed resume is retried by controller-runtime rather than
recreated from a missing timer.
@SyedaFatimaKazmi
SyedaFatimaKazmi merged commit fdd5012 into stakater:master Oct 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Deployment left permanently paused if resume's Get call fails — stale timer entry disables the missing-timer recovery

3 participants