[Bug] Plan executor produces delete+update on same Deployment UID, then retries stale plan indefinitely

[tristal]

Summary

The plan executor in execplan.go can produce a reconciliation plan that contains both a "delete Deployment" and "update Deployment" step for the same k8s Deployment resource. The delete succeeds, then the update fails on a stale UID precondition. The controller retries the same plan indefinitely without recomputing from current
state.

Environment

• Controller image: v1.5.x (exact TBD)
• Kubernetes: EKS 1.30
• Temporal: self-hosted (Cassandra-backed)

What happened

A helm release changed the desired image tag while an existing version was being sunset. The controller's reconcile plan included:

1. deleting deployment for account-temporal-default-worker-17-235-1-9755 ✅
2. deleted worker resource on version sunset (HPA) ✅
3. updating deployment for account-temporal-default-worker-17-232-4-559b ✅
4. updating deployment for account-temporal-default-worker-17-235-1-9755 ❌

Step 4 failed:

StorageError: invalid object, Code: 4, Key: /registry/deployments/.../account-temporal-default-worker-17-235-1-9755,
ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: be213a24-..., UID in object meta: ""

The controller retried this exact plan for ~3 hours. Restarting the controller produced zero reconcile activity for this TWD — it didn't re-enqueue or recompute.

Impact

The Worker Deployment's current build (17.235.1-9755) had its k8s Deployment deleted but was never recreated. Zero pods existed for the current version. All new workflows routed to it were stranded.

Proposed fix

1. Plan generator should not emit both delete and update for the same Deployment UID in a single plan.
2. On plan execution failure, recompute the plan from current state rather than retrying the stale plan.

[lucianobosco]

We're hitting what looks like the same failure mode on v1.7.0 (this issue was filed against v1.5.x), though our exact error string differs — possibly the same root cause in the plan executor manifesting differently.

Environment

• Controller image: temporalio/temporal-worker-controller:v1.7.0
• Kubernetes: GKE
• Temporal: self-hosted

Symptom

The controller's reconcile loops every ~30s on the sunset/scale step of a drained version's Deployment, never completing. Recurring log line (names anonymized):

{"level":"info","msg":"scaling deployment","controller":"workerdeployment",
 "WorkerDeployment":{"name":"my-worker","namespace":"my-namespace"},
 "deploymentError":"got runtime.Object without object metadata: &ObjectReference{Kind:Deployment,Namespace:my-namespace,Name:my-worker-abc1234-de56,UID:00000000-0000-0000-0000-000000000000,APIVersion:apps/v1,ResourceVersion:,FieldPath:,}",
 "replicas":0}

Note the executor is operating on an ObjectReference (no object metadata / empty ResourceVersion) instead of the full Deployment — consistent with the stale/empty-UID object described here.

Difference from the original report (milder manifestation)

In our case the controller is stuck on a drained version (scaled to 0/0), not the current build — so unlike the original report, the current build's Deployment stays healthy and no workflows are stranded. The visible impact is that drained versions never get GC'd: temporal worker deployment describe accumulates dozens of drained version entries going back weeks, and their zero-replica Deployments are never deleted.

Manual cleanup isn't possible either — temporal worker deployment delete-version is rejected with ManagerIdentity 'temporal-worker-controller/...' is set and does not match, since the controller owns the deployment.

The proposed fix here (recompute the plan from current state on execution failure rather than retrying the stale plan) would cover this case too.