Skip to content

Cap k3d create_cluster retries at three attempts - #421

Open
SebTardif wants to merge 1 commit into
xlab-uiuc:mainfrom
SebTardif:fix/cap-k3d-create-retries
Open

Cap k3d create_cluster retries at three attempts#421
SebTardif wants to merge 1 commit into
xlab-uiuc:mainfrom
SebTardif:fix/cap-k3d-create-retries

Conversation

@SebTardif

Copy link
Copy Markdown

Stop unbounded retries in k3d.create_cluster.

Problem

K3D.create_cluster retries forever when k3d cluster create keeps failing:

p = subprocess.run(cmd)
while p.returncode != 0:
    logger.error('Failed to create k3d cluster, retrying')
    self.delete_cluster(name)
    time.sleep(5)
    p = subprocess.run(cmd)

Kind already aborts after a few create failures. #419 caps delete_cluster. If delete succeeds and create still fails (bad image, docker error), this loop never exits.

Change

Raise RuntimeError("Failed to create k3d cluster") after three create attempts (two delete-and-retry cycles). No other k3d formatting.

Validation

pytest acto/kubernetes_engine/test_k3d_create.py

  • Red (unfixed): the raise test hung (5s timeout) because the loop never ended.
  • Green: 2 passed (three failures raise; success on the second create).

Origin

The loop has been in 153ef757 / #155 (2022-08-05, Yuxuan).

Ref #413
Ref #419

create_cluster deleted and retried forever when k3d create kept
failing. Stop after three attempts, matching delete_cluster.

Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant