Repository navigation
Description
Bug description
When a network-isolated workload's setup fails before the MCP container is created, its toolhive-<name>-internal network and <name>-dns container stay behind. Removing the workload doesn't clean them up either. Each leaked network uses one of Docker's address pools (about 31 by default). Once enough have leaked, every new workload fails with "all predefined address pools have been fully subnetted".
Steps to reproduce
- Run a network-isolated workload in an environment where a setup step after DNS startup fails. Observed trigger: a Docker bridge network whose IPv6 subnet has no gateway, which makes bridge-gateway discovery fail:
failed to set up workload: failed to create container: failed to discover Docker bridge gateways: docker bridge network <id> has an invalid IPAM gateway - The workload goes to
error. Remove it withthv rm <name>(orDELETE /api/v1beta/workloads/<name>). docker network ls | grep <name>still showstoolhive-<name>-internal, anddocker ps -a | grep <name>-dnsstill shows the dnsmasq container.
Expected behavior
Either a failed setup cleans up what it created, or thv rm removes the workload's internal network and auxiliary containers even when the MCP container was never created.
Actual behavior
Neither happens.
-
Setup returns without cleanup. In
DeployWorkload, the internal network (client.go:260) and the DNS container (client.go:267) are created before:- gateway discovery (client.go:277);
- egress proxy setup;
- port-binding generation;
- MCP container creation.
If any of those steps fails,
DeployWorkloadreturns the error without removing what it already created, and its callers (pkg/runtime/setup.go,pkg/runner/runner.go:515) don't either. -
Removal skips the runtime when the MCP container is missing.
deleteContainerWorkload(manager.go:929-958) finds no MCP container and goes straight to state cleanup. It never calls the runtime'sRemoveWorkload(client.go:494), and that is the only path that removes the-dns/-egress/-ingresscontainers and the internal network.RemoveWorkloadalso takes those names from the MCP container, so it couldn't find them without it.
Failures after the MCP container exists are not affected, since removal then goes through RemoveWorkload.
Environment
- ToolHive:
mainat f87c47a - Docker Engine 29.9.2 (Rancher Desktop, macOS)
Additional context
Seen in a local e2e run (2026-11-03). Every isolated workload failed at gateway discovery, and after the specs removed them, all 26 toolhive-*-internal networks and 26 *-dns containers were still present.
Related: #7043 (removing a workload whose start is still in flight leaks its network: a different trigger with the same symptom), #7042 (e2e harness workaround for #7043).
🤖 Generated with Claude Code
Activity