feat!: restore stranded HPA-managed Deployments after pre-deployment migrations#55
Merged
Conversation
…migrations With CHARTREUSE_UPGRADE_BEFORE_DEPLOYMENT, chartreuse-upgrade skips start_pods() and relies on the deployment that follows to restore replicas. That never happens for HPA-managed Deployments: their chart omits spec.replicas, and an HPA whose target is at 0 replicas with minReplicas >= 1 is ScalingDisabled forever. Every migration left those workers stranded at 0 replicas (e.g. pythie-cayzn-staging-demo on 2026-07-07). stop_pods() now marks every Deployment it scales down with the wiremind.io/stopped-by annotation, and the new chartreuse-restore entrypoint (run as a post-deployment hook, e.g. ArgoCD PostSync) re-enables the HPAs and scales back up the annotated Deployments still at 0 replicas - and only those, so a Deployment deliberately scaled to zero is left alone. A restore failure fails the Job, so the deployment is marked failed instead of stranding workers silently. BREAKING CHANGE: the wiremind-kubernetes dependency is dropped (library being decommissioned). The subset chartreuse uses now lives in chartreuse.kubernetes_helper and chartreuse.utils.command, and depends only on the official kubernetes client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allows validating a change on a canary environment before merging: tag any branch commit with a PEP 440 dev version and the package is published. The version is derived from the tag itself (never the VERSION file, so a dev tag cannot accidentally publish a final version) and the workflow refuses any tag that is not vX.Y.Z.devN. Stable releases keep going through a published GitHub Release only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tag follows semantic versioning; the workflow translates it to the PEP 440 equivalent (X.Y.Z.devN) for the PyPI package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haoov
approved these changes
Jul 8, 2026
Found during the staging-demo canary: the restore Job ran correctly but logged nothing — main() never configured logging (the upgrade path gets it via the Chartreuse class). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Canary validated on pythie-cayzn-staging-demo (chartreuse 7.0.0.dev1 + chart 6.4.0-dev.1): real alembic migration ran on the upgradeBeforeDeployment path — stop_pods stopped+annotated the workers at 15:04, PostSync chartreuse-restore brought the HPA-managed workers back at 15:07 and the HPAs re-armed and enforced their minimums. Without this fix those workers stayed at 0 forever (2026-07-07 incident). One fix found during the canary and already pushed here: configure logging in the restore entrypoint (a4ff120). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
CHARTREUSE_UPGRADE_BEFORE_DEPLOYMENT,chartreuse-upgradeskipsstart_pods()and relies on the deployment that follows to restore replicas. That never happens for HPA-managed Deployments: their chart omitsspec.replicas(so ArgoCD/Helm applies never touch it), and an HPA whose target is at 0 replicas withminReplicas >= 1isScalingDisabledforever. Every migration leaves those workers stranded at 0 replicas — this hitpythie-cayzn-staging-demoon 2026-07-07 (3 worker deployments down for ~24h), and applies to every env withupgradeBeforeDeployment: true, including all pythie prods.Fix
stop_pods()now marks every Deployment it scales down with thewiremind.io/stopped-byannotation. Live-only annotations survive ArgoCD sync/selfHeal (per-key field ownership), which is exactly the mechanism that orphanedspec.replicasin the first place.chartreuse-restoreentrypoint, run as a post-deployment hook (ArgoCD PostSync — see companion chart PR): re-enables HPAs and scales back up annotated Deployments still at 0 — and only those, so a Deployment deliberately scaled to zero by a human is left alone. Any replicas >= 1 re-arms the HPA, which then enforces its own minReplicas.BREAKING CHANGE
The
wiremind-kubernetesdependency is dropped (library being decommissioned). The subset chartreuse used (deployment manager, kube config loading,run_command) now lives inchartreuse.kubernetes_helper/chartreuse.utils.command, depending only on the officialkubernetesclient. Version bumped to 7.0.0 (ensure_safe_runrequires the chart's appVersion to match major.minor).Companion PR: wiremind/wiremind-helm-charts (chartreuse chart 6.4.0)
🤖 Generated with Claude Code