Skip to content

Failed workload setup leaves its network and DNS container behind #7045

Open

Description

@tgrunnagle

Bug description

When a network-isolated workload's setup fails before the MCP container is created, its toolhive-<name>-internal network and <name>-dns container stay behind. Removing the workload doesn't clean them up either. Each leaked network uses one of Docker's address pools (about 31 by default). Once enough have leaked, every new workload fails with "all predefined address pools have been fully subnetted".

Steps to reproduce

  1. Run a network-isolated workload in an environment where a setup step after DNS startup fails. Observed trigger: a Docker bridge network whose IPv6 subnet has no gateway, which makes bridge-gateway discovery fail:
    failed to set up workload: failed to create container: failed to discover Docker bridge gateways: docker bridge network <id> has an invalid IPAM gateway
    
  2. The workload goes to error. Remove it with thv rm <name> (or DELETE /api/v1beta/workloads/<name>).
  3. docker network ls | grep <name> still shows toolhive-<name>-internal, and docker ps -a | grep <name>-dns still shows the dnsmasq container.

Expected behavior

Either a failed setup cleans up what it created, or thv rm removes the workload's internal network and auxiliary containers even when the MCP container was never created.

Actual behavior

Neither happens.

  • Setup returns without cleanup. In DeployWorkload, the internal network (client.go:260) and the DNS container (client.go:267) are created before:

    • gateway discovery (client.go:277);
    • egress proxy setup;
    • port-binding generation;
    • MCP container creation.

    If any of those steps fails, DeployWorkload returns the error without removing what it already created, and its callers (pkg/runtime/setup.go, pkg/runner/runner.go:515) don't either.

  • Removal skips the runtime when the MCP container is missing. deleteContainerWorkload (manager.go:929-958) finds no MCP container and goes straight to state cleanup. It never calls the runtime's RemoveWorkload (client.go:494), and that is the only path that removes the -dns/-egress/-ingress containers and the internal network. RemoveWorkload also takes those names from the MCP container, so it couldn't find them without it.

Failures after the MCP container exists are not affected, since removal then goes through RemoveWorkload.

Environment

  • ToolHive: main at f87c47a
  • Docker Engine 29.9.2 (Rancher Desktop, macOS)

Additional context

Seen in a local e2e run (2026-11-03). Every isolated workload failed at gateway discovery, and after the specs removed them, all 26 toolhive-*-internal networks and 26 *-dns containers were still present.

Related: #7043 (removing a workload whose start is still in flight leaks its network: a different trigger with the same symptom), #7042 (e2e harness workaround for #7043).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageIssue needs initial triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions