Skip to content

fix(hpa): pause HPAs via scaleUp selectPolicy instead of breaking scaleTargetRef#56

Merged
arthlr merged 1 commit into
mainfrom
chartreuse-hpa-pause
Jul 9, 2026
Merged

fix(hpa): pause HPAs via scaleUp selectPolicy instead of breaking scaleTargetRef#56
arthlr merged 1 commit into
mainfrom
chartreuse-hpa-pause

Conversation

@arthlr

@arthlr arthlr commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Problem

disable_hpa() disables an HPA during migrations by renaming its scaleTargetRef to a non-existent wm--disabled--kube-* Deployment. Two defects:

  • The HPA controller emits FailedGetScale / AbleToScale=False, which ArgoCD's HPA health check reads as Degraded — every migration flips the app to Degraded (observed fleet-wide during the 2026-07-08 rollout), training humans and alerting to ignore Degraded and health-gating ArgoCD wave progression/PostSync hooks.
  • Under GitOps the rename is unreliable: ArgoCD owns scaleTargetRef and reverts it on any sync.

Fix

Pause with the native mechanism instead: spec.behavior.scaleUp.selectPolicy: Disabled. The HPA stays Healthy (no error events, no Degraded) but cannot scale up — the only thing a migration needs to prevent. The pre-pause spec.behavior is saved in the wiremind.io/pre-pause-scale-behavior annotation and restored exactly on resume (or the field is cleared when there was none, avoiding server-defaulted leftovers).

Safety trade-off handled: unlike the rename (accidentally healed by GitOps), a leftover pause is invisible to ArgoCD under server-side apply and would linger after a crashed run — so start_pods() and restore_stopped_pods() now resume HPAs unconditionally for every EDS-tracked Deployment (resuming an unpaused HPA is a no-op). The rename-repair path (re_enable_hpa) stays one more minor for HPAs left broken by a pre-7.0.1 run — TODO removal in 8.0.

Versioning

7.0.1 as a fix: no chart contract change (ensure_safe_run compares major.minor, chart appVersion stays 7.0.0), consumers pick it up through chartreuse~=7.0.0. No release train needed.

Validation

  • 100 unit tests pass (pause idempotence, exact behavior restore, unconditional resume, rename-repair shim).
  • Canary rails available if wanted: tag v7.0.1-dev.1 → PyPI dev package → staging-demo migration; expected observation: app stays Healthy during migration, pause annotation + scaleUp: Disabled visible on HPAs while the job runs, both gone after PostSync, zero FailedGetScale events.

🤖 Generated with Claude Code

…leTargetRef

Renaming the scaleTargetRef to a non-existent Deployment made the HPA emit
FailedGetScale and turn the ArgoCD application Degraded for the whole
migration window, and was reverted by any GitOps sync anyway.

Pause instead with spec.behavior.scaleUp.selectPolicy: Disabled - the HPA
stays Healthy but cannot start pods mid-migration, which is all we need.
The pre-pause spec.behavior is saved in the
wiremind.io/pre-pause-scale-behavior annotation and restored exactly on
resume.

A leftover pause is not cleaned up by GitOps (the live-only field is
invisible to ArgoCD under server-side apply), so start_pods() and
restore_stopped_pods() resume HPAs unconditionally for every tracked
Deployment. The rename-repair path (re_enable_hpa) is kept one more minor
for HPAs left broken by a pre-7.0.1 run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arthlr arthlr self-assigned this Jul 9, 2026
@arthlr arthlr requested a review from haoov July 9, 2026 09:39
@arthlr arthlr merged commit cb5ed8f into main Jul 9, 2026
5 checks passed
@arthlr arthlr deleted the chartreuse-hpa-pause branch July 9, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants