FlowpyterOperator leaks Docker containers (stuck in Created) when a task is interrupted

[jakejellinek]

Summary

In production (drc-srv-01, flowpyter-task 1.4.1) the flowbot aggregate DAGs steadily accumulate dormant Docker containers stuck in Created state — ~48 built up over roughly two months, mostly busybox prep containers plus the occasional papermill container. They hold no state and do nothing, but they clutter docker ps -a and make incident triage noisier. Root cause: container removal only happens on the happy path, so any interruption leaks a container.

Mechanism

PapermillOperator.execute() (src/flowpytertask/flowpytertask.py:390) calls create_path_on_host() twice. When the operator runs inside a container, that spawns a busybox helper:

# src/flowpytertask/flowpytertask.py:370
client.containers.run(
    "busybox",
    f"mkdir -p /opt/{relative_path}",
    volumes=[f"{host_root}:/opt"],
    user=self._user_string,
    remove=True,          # :375
)

and the papermill container itself is created by the parent DockerOperator with auto_remove="force" (:229).

Both cleanup paths only fire on normal completion:

• busybox — remove=True (docker-py) removes the container after the synchronous run() returns. If the worker process is SIGKILLed, or create succeeds but start blocks/fails (e.g. the Docker API 60 s timeout under host disk saturation — see the sibling deployment's notes), the removal never runs and the container is left in Created.
• papermill — auto_remove="force" is daemon-side and only triggers when the container exits. One created but never started (task killed in the create→start window) never exits, so it lingers in Created.

Neither container is tracked anywhere on_kill can reach — the busybox container isn't stored on the operator, so DockerOperator.on_kill (which stops/removes self.container, the papermill one) can't clean it up. The cod-crisis-aggregates DAG retries 3× (waiting a day for late CDR data) and runs under a concurrency pool, so interruptions — and therefore leaks — are routine rather than exceptional.

Impact

Operational/cosmetic only — no data at risk, the leftover containers are inert. But accumulation is unbounded without external cleanup.

Suggested fixes (any of, roughly in order of robustness)

1. Avoid the container where possible. create_path_on_host already has a non-container branch ((host_root / relative_path).mkdir(exist_ok=True, parents=True) when is_running_on_docker() is false). If the worker can create the path directly with the right uid/gid, skip the busybox spawn entirely.
2. Track + guarantee cleanup of the prep container. Replace containers.run(remove=True) with explicit create / start / wait and remove it in a finally, storing its id on self so on_kill also removes it on interruption.
3. Label the helper containers (e.g. labels={"flowpyter-task": "prep"}) so orphans are unambiguously identifiable and can be reaped precisely instead of by a blunt status=created filter.

Current mitigation

On drc-srv-01 we added an hourly systemd timer that removes only Created-state containers older than 1 h (safe by construction: it cannot race a task that is still launching, and never touches running/exited containers). See Flowminder/drc-srv-01-flowkit-deployment PR #13. This is a per-deployment workaround; a library-level fix would benefit every flowpyter-task deployment.

Environment

• flowpyter-task 1.4.1 (line refs above match current main: 229, 370, 375, 390)
• Containerised Airflow worker; PapermillOperator → Airflow DockerOperator base