feat(demo): let teardown.sh undo the OPA install and delete local images

[omerboehm]

Summary

After a full onboarding demo run, ./teardown.sh still left parts of the demo on the cluster: bundle-service, the local authbridge image pin, and every localhost/* image. Cleaning them up meant extra manual steps. This PR adds that cleanup to the script behind opt-in flags.

• --include-opa now undoes everything k8s/opa-kind-enable.sh installed:
  1. Runs opa-kind-restore.sh to remove the OPA legs from the pipeline, as before.
  2. Deletes bundle-service: its Deployment, Service, ServiceAccount, ClusterRole/Binding and the default AuthorizationPolicy. It was applied with helm template | kubectl apply, so nothing else ever removes it.
  3. Drops the localhost/authbridge:local pin from the rossoctl release and restarts the operator so new sidecars use the operator subchart's default image. The script then checks the rendered rossoctl-platform-config.
• --include-images deletes the locally built images from the Kind node(s) and the host runtime. It always covers the four AIAC stack images, github-agent, github-tool and keycloak-aiac. It also covers operator and authbridge, but only together with --include-opa. Any image a running pod still uses is kept and reported.
• --all combines both flags.
• The survey step now shows whether bundle-service is present and which authbridge image is configured.
• demo.md Cleanup section updated to match.

The AuthorizationPolicy CRD is still left in place, because deleting it would delete every policy CR on the cluster.

Why the pin is dropped with -f

helm upgrade --reuse-values --set operator-chart.defaults.images.authbridge=null removes the key from the stored values, but the release still renders localhost/authbridge:local into rossoctl-platform-config, even after a second --reuse-values upgrade. Passing the stored values back explicitly with -f renders the chart default. It's done with JSON from helm get values -o json, so PyYAML isn't needed.

Test plan

• bash -n teardown.sh
• ./teardown.sh --dry-run --all on a Kind cluster: all new steps are listed. Pin detection correctly reports the chart default.
• ./teardown.sh --dry-run --include-images: operator and authbridge are not listed without --include-opa.
• An unknown flag prints the updated usage line.
• The bundle-service deletion, the -f helm upgrade, the operator restart and the image removal were run by hand on the same cluster and worked.
• Not yet run by the script itself: a real ./teardown.sh --all after a full demo run. The cluster was already clean, so only the dry run could be exercised.

🤖 Generated with Claude Code

[oblinder]

Automated code review (Claude Code). 10 findings, all in teardown.sh. 9 are inline comments below.

Not inline (line 234 is outside the diff): confirmation prompt is incomplete. --aiac-only together with --include-images is accepted, but the prompt says "Nothing else is touched" and then images are deleted from the node and the host. Also, the non---aiac-only prompt does not say that --include-opa runs a helm upgrade on the platform release and restarts the operator, or that --include-images deletes images. A user who runs --all approves a platform helm upgrade that the prompt does not mention.

🤖 Generated with Claude Code

[oblinder]

If drop_authbridge_pin fails, this warning says the authbridge image "will be kept below too", but no code keeps it. The image is kept only when a pod is running it at that time.

Failure: on --all, the helm upgrade (or helm get values) fails. restore.sh has already deleted github-agent, and no other agent pod runs the sidecar. So IMAGES_IN_USE does not contain localhost/authbridge:local, and remove_image runs crictl rmi on it. The operator still injects that image, which cannot be pulled, so all new agent pods go to ErrImagePull/ImagePullBackOff.

Fix: when the drop fails, add the image to a keep list.

[oblinder]

When opa-kind-restore.sh fails, the script only warns and continues. It then deletes bundle-service and the default AuthorizationPolicy, so the OPA legs stay wired with no bundle source.

Failure: the restore fails (for example, on a helm dependency build or upgrade error). The OPA plugin stays in authbridge-runtime-config in team1, but the next step deletes bundle-service. All OPA-gated requests from the remaining team1 agents then fail closed. This is worse than the state before the teardown.

Fix: skip the bundle-service delete and the pin drop when the restore fails.

[oblinder]

drop_authbridge_pin restarts only the operator. Agent pods that are already running keep the injected localhost/authbridge:local sidecar, but the function reports that the sidecar image is back to the chart default.

Failure: opa-kind-restore.sh deletes the agent pods while the pin is still applied, so they are recreated with the local sidecar. This function then re-renders the release and restarts the controller-manager, but does not restart those pods. Non-demo agents in team1 keep the local image, --include-images keeps it as "still used", and the pass is wrong.

Fix: after the operator restart, delete or restart the agent pods (-l rossoctl.io/type=agent).

[oblinder]

If kubectl get pods -A fails, 2>/dev/null || true makes IMAGES_IN_USE silently empty. All images are then treated as unused and deleted, including images that pods are running.

Failure: a transient API error, or RBAC that refuses list on all namespaces. remove_image then runs crictl rmi on images in use (for example, a keycloak-aiac:local that restore.sh failed to revert), and that workload cannot restart (ImagePullBackOff).

Fix: fail, or skip image removal, when the pod listing fails.

Also on this line: the list includes pods that are still Terminating (the github-agent/github-tool pods that restore.sh just deleted, and aiac-system pods when the namespace delete times out). Their images are kept as "still used by a running pod", and the next enable reuses the stale image. Fix: filter out pods that have a deletionTimestamp, or wait for the pods to go.

[oblinder]

remove_image ignores every failure of crictl rmi and image rm, but this line always prints "local images removed".

Failure: a stopped host container still references localhost/aiac-agent:local, so docker image rm fails. Or crictl rmi fails, or CONTAINER_RUNTIME is the wrong runtime. The script prints nothing about the failure and reports success. A later ./enable.sh still finds the stale image.

Fix: count the failures and warn instead of pass when any removal fails.

[oblinder]

--include-images needs kind to find the nodes, but preflight does not check that kind is on PATH. When kind is missing, || true hides the error, KIND_NODES is empty, and only the host copies are removed. The node-side images (the ones that hold disk and make enable.sh skip the rebuild) stay, but the summary says the images were removed.

Fix: add a command -v kind check to preflight when DO_IMAGES=1.

[oblinder]

The operator and authbridge image names are hard-coded here, and the pin check in drop_authbridge_pin matches only ^localhost/. The OPERATOR_IMAGE / IMAGE_TAG overrides that opa-kind-enable.sh accepts are ignored.

Failure: the user ran opa-kind-enable.sh with IMAGE_TAG=localhost:5000/authbridge:dev. The post-upgrade check misses that image, and --include-images never deletes the images that were actually built. The teardown reports a full reset while those images remain.

Fix: read the same overrides, or read the pinned image from helm get values.

[oblinder]

This block is a third copy of the CONTAINER_RUNTIME detection in enable.sh and opa-kind-enable.sh, and its fallback is different: it selects podman when docker is missing, even if podman is also missing. The other two scripts select podman only when podman exists.

On a host with neither runtime on PATH, this script selects podman and every remove_image call fails silently. A shared helper in a common lib would keep all three copies the same.