Skip to content

ci: Add opt-in bootc-operator reverse dependency test - #2

Draft
cgwalters-bot wants to merge 1 commit into
mainfrom
bot/operator-revdep
Draft

cgwalters-bot wants to merge 1 commit into
mainfrom
bot/operator-revdep

Conversation

@cgwalters-bot

@cgwalters-bot cgwalters-bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

bootc-operator drives bootc switch/bootc upgrade on Kubernetes nodes, so a bootc change can break it in ways our own tests don't cover. This adds an opt-in workflow that runs the operator's e2e suite against bootc built from the PR.

How it works: the operator's e2e tests boot bink node VMs from a prebuilt disk image. The workflow builds Fedora 44 RPMs from the tree (just package), layers them onto the same bink node image the operator tests with (read from the label of the operator's default disk image), round-trips it through the bink registry so its digest matches what the nodes see, and builds a disk image from it with bink's own build-disk-image (so the PR's bootc also does the install to-disk). It then runs the operator's own deploy-bink and e2e targets, skipping seed-node-image with make -o since we seeded the registry ourselves. The bink commit comes from the operator's CI workflow, so the two stay in step, and the host setup mirrors the operator's e2e job. No change to bootc-operator is needed.

It takes roughly 1.5 hours, so it's opt-in: for PRs labeled ci/operator (on the label being added, and on later pushes while it's present), via workflow_dispatch (with an optional operator ref), and weekly on bootc-dev. A maintainer needs to create the ci/operator label.

The catch is that this is coupled to internals of the operator (its deploy-bink/e2e/seed-node-image Makefile targets and print-var-* helper, the localhost:5000/node:latest registry convention, BINK_COMMIT in its CI workflow) and of bink (its node-images/fedora build-disk-image target). If either side reshuffles those, this breaks, as the header comment in the workflow says. A more robust follow-up would be for bootc-operator to expose its e2e job as a reusable workflow_call workflow taking an RPM artifact (or node image), so the knowledge of how to test it lives in one place and this file shrinks to a single uses:.

FYI: bink's build-disk-image currently builds bcvk from a personal fork branch, Johan-Liebert1/bcvk@img-to-install-cli-opt (a workaround for a digest mismatch, see bootc-dev/bcvk#314, still open), so the install path here also goes through that branch until bink switches back to upstream bcvk.

Design choices that came up in review, kept deliberately:

  • The label is ci/operator, following the existing ci/* label naming.
  • The weekly schedule stays: it's the only thing that notices drift on the operator/bink side when no one has labeled a PR. Note scheduled-run failures only get emailed to whoever last edited the cron line; if that's not enough, a follow-up could open an issue on failure.
  • The package job isn't cached or shared. The better dedup would be to consume ci.yml's packages artifact once that's practical, rather than adding a separate cache here.

Lessons from composefs-rs's bootc-revdep.yml (which records a failed run on every push because of an invalid if:): the gating here uses only valid expressions and actionlint is clean apart from the ubuntu-26.04 runner label, which it doesn't know yet and which ci.yml already uses. Unlabeled PRs get a skipped run, not a failed one.

Testing:

  • The workflow itself ran for real on a PR inside the cgwalters-forge fork, triggered by adding ci/operator: https://github.com/cgwalters-forge/bootc/actions/runs/35929518061 passed. The node image picked up the PR's RPM (bootc-202609232239.g43ff95f900), and all 9 operator e2e tests passed (ok .../test/e2e 1925s). Wall time: 30 min for package (no build cache), 55 min for e2e, of which 10 min is the disk image build.
  • After review fixes (concurrency grouping, the installed-RPM assertion, package timeout and persist-credentials: false), a second full run at the current head also passed: https://github.com/cgwalters-forge/bootc/actions/runs/35937468141. The new assertion held (rpm -q bootc = bootc-202609240013.g5c07bd6a3f-1.fc44, the PR build, replacing the stock bootc-202609220256.g5ed9e3f700), and all 9 e2e tests passed (ok .../test/e2e 1915s). Adding an unrelated label (ci/merge) while it was running produced its own skipped run (https://github.com/cgwalters-forge/bootc/actions/runs/35937496687) and did not cancel it.
  • On the same fork PR, a push without the label gave a skipped run, and the opened event gave none.
  • The same steps replayed by hand (as a script) on a 16-core RHEL 10 devspace: just package for fedora-44, node and disk image build, make -o seed-node-image deploy-bink, then make e2e, with all 9 tests passing in 1328s. That also confirmed that make -o skips the phony seed-node-image target and that TestControllerMembership's booted-digest check passes with the round-tripped image.
  • actionlint is clean apart from the ubuntu-26.04 label, same as ci.yml.
  • Not exercised: workflow_dispatch (including a non-default operator-ref) and the weekly schedule. After the test run, the only change was the duration wording in a comment.

CI note (2026-09-24): the test-integration (centos-10, composefs, xfs, systemd, bls, unsealed) failure is unrelated to this change: plan-44-shadow-fixup hit a guest reboot timeout because bootc-finalize-staged.service timed out stopping (the /boot automount expire deadlock), the known flake fixed by #6. Failed jobs will be rerun once the run completes.

Also note: the labeled event also re-runs ci.yml (it already lists labeled), which predates this change.

Closes: bootc-dev#2423

Generated-by: https://github.com/cgwalters/#llms


Review draft in cgwalters-forge, not upstream yet. This section is removed when the PR is opened upstream.

  • Upstream: bootc-dev/bootc, base main
  • Board item: PVTI_lADOE9oHIs4BlJLczg9kELo
  • Fork CI: off; the devspace testing described above is this PR's CI, and upstream CI runs once it is opened there

To review:

  • Approve, or comment /promote on a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.
  • If upstream requires DCO, approving also signs off: promote adds Signed-off-by: Colin Walters <walters@verbum.org> to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer.
  • Add a /draft line (in the same comment or before) to open it upstream as a draft (/ready undoes that).
  • Close to drop it.
  • Edit the title and description freely: they become the upstream PR's. Review comments are squashed into the commits they concern, with a reply here.

@cgwalters-bot cgwalters-bot added ci/operator Run bootc-operator e2e revdep test and removed ci/operator Run bootc-operator e2e revdep test labels Sep 23, 2026
@cgwalters-bot
cgwalters-bot force-pushed the bot/operator-revdep branch 2 times, most recently from c1cd8ab to 5f3b33d Compare September 24, 2026 00:08
@cgwalters-bot cgwalters-bot added the ci/operator Run bootc-operator e2e revdep test label Sep 24, 2026
bootc-operator drives `bootc switch`/`bootc upgrade` on Kubernetes
nodes, so a bootc change can break it in ways our own tests don't
cover. Run its e2e suite against bootc built from this tree.

The operator's e2e tests boot bink node VMs from a prebuilt disk
image, so we layer our RPMs onto the same bink node image the operator
uses, build a disk image from it with bink's own tooling (which also
means the PR's bootc does the install), and seed the bink registry
with it in place of the operator's `seed-node-image` step. Everything
else is the operator's own Makefile targets and host setup, with the
bink commit taken from the operator's CI so the two stay in step.

It takes roughly 1.5 hours and depends on another repository, so it only
runs for PRs with the `ci/operator` label, on demand, and weekly.

Closes: bootc-dev#2423
Generated-by: AI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/operator Run bootc-operator e2e revdep test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

run operator CI (at least optionally)

1 participant