Skip to content

ci: run tests and release on merge to stable through the shared workflows - #120

Open
singlerider wants to merge 15 commits into
devfrom
ci/publish-on-tag
Open

singlerider wants to merge 15 commits into
devfrom
ci/publish-on-tag

Conversation

@singlerider

@singlerider singlerider commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Moves blasthttp CI onto the org's shared workflows. A release is a reviewed merge into stable that bumps the version, approved at the release environment.

Shared workflows

Calls the reusable workflows in blacklanternsecurity/CLA, pinned to CLA#11 at 07e14bcbba5e737e1b29bd9e3f9128fbf2483be9. Re-pin to its merge commit before this merges.

  • tests.yml (pull_request, workflow_call) calls CLA's python-tests.yml. Until stable's ruleset moves, a required job reports python-tests (3.11) to (3.15) from the real result. rust-tests.yml keeps its live rust-tests name, now with every action SHA-pinned, contents: read, and persist-credentials: false. Neither triggers on push: stable any more, and their concurrency groups are separate from the calls in publish.yml.
  • publish.yml (push to stable) calls release-check.yml, both test workflows, maturin-wheels.yml (linux and musllinux, vendored OpenSSL), publishes to PyPI with OIDC in release, then crates.io through crates.yml with the id-token: write it needs (without it the whole run failed at startup), then publish.yml (tag and GitHub release with SBOM and wheels, created last). No caller passes App credentials.
  • stable-guard.yml: release ready on pull requests into stable refuses a non-MAJOR.MINOR.PATCH version and git or URL dependencies.
  • cla.yml passes APP_ID/APP_PRIVATE_KEY by name. No workflow uses secrets: inherit.
  • Version 0.10.2: 0.10.1 is on PyPI and crates.io with no v0.10.1 tag. pyproject.toml reads it from Cargo.toml.
  • release.toml (cargo-release, which pushed v{{version}} tags) removed. README ## Releasing links the public org release standard.
  • Merges current dev (AGPL-3.0, Python 3.15 support).

Before promoting to stable

  • release environment with reviewers and a stable-only deployment policy, holding BLS_RELEASE_APP_ID and BLS_RELEASE_PRIVATE_KEY.
  • Trusted publishers on PyPI and crates.io for publish.yml, environment release. Then delete PYPI_API_TOKEN.
  • Ruleset 14658129, with .github-config#1/Fix CI: add perl-List-Util for manylinux, distro-detect for musllinux… #2: require python / passed, rust-tests and release ready, then drop the required job.

Validation

$ uvx --from actionlint-py actionlint
(no findings)
$ grep -rhn 'uses: ' .github/workflows | grep -vE '@[0-9a-f]{40}|uses: \./'
(none)
$ cargo metadata --locked --no-deps
(lock consistent at 0.10.2)

Refs blacklanternsecurity/bbot-enterprise#139, blacklanternsecurity/bbot-enterprise#140, blacklanternsecurity/bbot-enterprise#149, blacklanternsecurity/bbot-enterprise#155, blacklanternsecurity/bbot-enterprise#156 (blasthttp part)

Move PyPI (maturin) and crates.io publishing out of python-tests.yml
and rust-tests.yml into publish.yml, triggered only by v* tags.

- version job fails unless the tag equals v + the Cargo.toml version
- test workflows run via workflow_call; every build job gates every
  registry upload so a release is never partial
- drop --skip-existing and registry already-published probes
- pyproject reads its version from Cargo via dynamic = ["version"]
- concurrency groups use a fixed per-file prefix since github.workflow
  is the caller's name under workflow_call
- pin every action to a full SHA, replace -latest runners

Part of blacklanternsecurity/bbot-enterprise#139
Part of blacklanternsecurity/bbot-enterprise#149
Part of blacklanternsecurity/bbot-enterprise#156
The publish workflow now refuses any tag that is not vMAJOR.MINOR.PATCH
or vMAJOR.MINOR.PATCH-rc.N before comparing it to Cargo.toml, so the
org tag convention is enforced rather than implied.

PyPI upload drops PYPI_API_TOKEN and relies on trusted publishing via
the job's id-token permission. Registry jobs run in the release
environment so the publish gate applies.

A final job creates the GitHub Release for the tag in the same run and
attaches an SPDX JSON SBOM from anchore/sbom-action, so a tag and its
release always exist together.

CLA calls the shared reusable workflow instead of carrying a fork.
Governance declares one gated environment named pypi across published
repos. crates and docker jobs stay in release.
@singlerider
singlerider requested a review from en0f October 3, 2026 13:59
@singlerider singlerider self-assigned this Oct 3, 2026
rust-tests.yml, python-tests.yml and the inline wheel matrix, crates
upload and release steps duplicated what every PyO3 repo carries, and
the wheel builds hardcoded the interpreter list. tests.yml now calls the
shared rust-tests.yml and python-tests.yml in blacklanternsecurity/CLA
with scripts/build-openssl.sh as setup. publish.yml calls
release-check.yml, maturin-wheels.yml (Linux families only, the
container OpenSSL build now lives in scripts/wheel-container-setup.sh),
crates.yml and publish.yml, keeping only the trusted-publishing PyPI
upload at top level.

pyproject drops managed = false and adds maturin to the dev group so
the shared workflow can uv sync and maturin develop. release.toml is
removed: cargo-release competed with the one release cut tool. README
examples are ruff formatted, which ruff format --check enforces.
Completes the previous commit, which removed the old test workflows but
missed the files replacing them. tests.yml calls the shared rust and
python test workflows with scripts/build-openssl.sh as setup. publish.yml
calls release-check, maturin-wheels (Linux families, container OpenSSL
build in scripts/wheel-container-setup.sh), crates and publish, keeping
only the trusted-publishing PyPI upload at top level. pyproject lets uv
manage the project and adds maturin to the dev group. README examples
are ruff formatted.
Releases are a pushed tag cut with the shared release.sh, which checks
the tag against the manifest. State that here instead of copying the
script.
@singlerider singlerider changed the title ci: publish only on version tags ci: run tests and tag-only publish through the shared workflows Oct 3, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. PRs into stable will be stuck. The default ruleset on stable requires checks named rust-tests and python-tests (3.10) through (3.14). This PR deletes the workflows that produce those checks. The new shared workflows report under different names (rust / passed, python / passed, python / test (3.x), and so on). Once this reaches dev, the next dev → stable PR will wait forever on checks that never run. Before merging, change the required checks in that ruleset to rust / passed and python / passed. The dev ruleset has no required checks, so this PR itself can merge.

  2. The pins point at a branch that hasn't been merged. publish.yml and tests.yml pin blacklanternsecurity/CLA@e532142…, a commit on feat/shared-workflows. That CLA PR is still open. If it's squash-merged and the branch is deleted, that commit may stop being fetchable, and tests and releases would fail to resolve the workflows. Merge the CLA PR first, then update the pins (and the README link) to the matching commit on CLA main.

Check before the first tag: the PyPI job now publishes with trusted publishing (environment: pypi) instead of PYPI_API_TOKEN. Unless PyPI already has a trusted publisher registered for blacklanternsecurity/blasthttp, workflow publish.yml, environment pypi, the first release will fail at the PyPI step. I can't see PyPI settings from here, so I couldn't confirm it.

@liquidsec liquidsec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not remove the rust tests here. They have highly customized tests that will not apply to any other repository (for example, custom openssl build)

Restore rust-tests.yml from dev and drop the shared rust job from
tests.yml. The Rust job builds a custom OpenSSL via
scripts/build-openssl.sh, caches it, and runs fmt, clippy and tests
in a way that is specific to this repo. The rust-tests job name is
kept since it is a required status check.

The rust-publish job stays removed: crates.io publishing now runs
only on version tags via publish.yml.
rust-tests.yml is now callable and publish.yml runs it alongside
tests.yml before building wheels, so a tag cannot release a crate or
wheel whose Rust tests, clippy, or fmt fail.

The concurrency group is named explicitly and only cancels on pull
requests, matching tests.yml. Inherited github.workflow would resolve
to the caller's name and collide with the publish run's own group.
@singlerider
singlerider requested review from en0f and liquidsec October 6, 2026 14:33
CLA#3 was squash-merged into main. The pinned workflow files and
scripts/release.sh are byte-identical at de82726743e51ac51a1be31196d893312a9fea59.

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two blocking issues: one is already live, the other will hit as soon as Python 3.15.0 ships. Everything else checks out.

  1. PRs into stable will hang (still open from your earlier review). The default ruleset on stable still requires python-tests (3.10) through (3.14). This PR deletes python-tests.yml, and the shared workflow reports those results under different names (python / test (3.x) and python / passed), so the old check names never appear. Every dev → stable PR would wait on them forever. rust-tests is fine because the repo's own workflow is kept. Fix: swap those five required checks for python / passed before this lands.

  2. Python 3.15 will break PR CI and the wheel builds.

How: The shared python-versions.yml picks the Python versions from requires-python = ">=3.10" and whatever versions uv can download. Once uv ships 3.15.0 final, 3.15 joins both the test matrix and the wheel -i list.
Why it fails: The locked PyO3 (pyo3-ffi 0.27.2) caps CPython at 3.14. Both old workflows set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to get past that cap, and the new calls don't pass it.
What breaks: python / test (3.15) fails, which fails python / passed on every PR, and the publish run fails at wheels.
Timing: It isn't failing yet, because the latest uv (0.12.23) still lists only 3.15.0rc3, which the shared script skips.
Fix: In tests.yml and in the wheels job of publish.yml, pass env: '{"PYO3_USE_ABI3_FORWARD_COMPATIBILITY": "1"}'. Bumping PyO3 to a release that supports 3.15 would also work.

The locked pyo3-ffi 0.27.2 caps CPython at 3.14. The removed
workflows set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1, and the shared
python-tests and maturin-wheels callers did not, so 3.15 would fail
the test matrix and the wheel builds once uv ships it.

Pass the variable through the env input of both calls.
@singlerider
singlerider requested a review from en0f October 6, 2026 18:31
en0f
en0f previously approved these changes Oct 6, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but @singlerider see comment on repo settings:

PRs into stable will hang. I checked the live default ruleset on stable. It still requires these checks:

rust-tests (still produced, because the repo keeps its own workflow)
python-tests (3.10) through python-tests (3.14)

This PR deletes python-tests.yml. The shared workflow reports under python / test (3.x) and python / passed instead. Once this reaches dev, the next dev → stable PR will wait forever on five checks that never run. The fix is in repo settings, not the code: swap those five required checks for python / passed before or right as this merges. The dev ruleset has no required checks, so this PR itself can merge.

The shared workflow pins pointed at de82726 (and the CLA caller at
4f85d3c where it still lagged). CLA main is now 40c6e18, which adds the
optional registry login for image SBOMs. Every CLA reference, including
the release.sh link, now resolves on CLA main at one commit.

Refs blacklanternsecurity/bbot-enterprise#139

@liquidsec liquidsec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will mean anyone with write access can publish. This needs to be discussed.

Follows .github-config#17 through CLA#11: publish.yml runs on push to
the trunk instead of a v* tag, publishes in the release environment a
bbot-leads member approves, re-runs safely (PyPI skip-existing,
crates.io only when the version is new), and leaves the tag and the
GitHub release to the bls-release App at the end of the run. The
pypi environment and the long-lived registry token are gone.
@singlerider

Copy link
Copy Markdown
Contributor Author

@en0f @liquidsec reworked to the approved publishing policy (.github-config#17), through CLA#11. This answers "anyone with write access can publish" and the review and pipeline gates:

  • publish.yml runs on a merge to the trunk, not on a tag. A release is a reviewed PR into the trunk that bumps the version.
  • Every publishing job runs in the release environment, which a bbot-leads member other than the run's starter must approve; admins can't skip it.
  • Tests run before anything publishes. Re-runs are safe: PyPI skip-existing, crates.io only if the version is new.
  • The tag and GitHub release come last, from the bls-release App. The tag rulesets in .github-config#1 block everyone else.
  • The pypi environment and the long-lived registry tokens are gone.

@singlerider
singlerider requested review from en0f and liquidsec October 21, 2026 12:55

@pjhartlieb pjhartlieb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@singlerider, I reviewed this at 349e58d5 (based on current dev, a9c04545) against the .github-config#16 spec part B, the publishing policy, and CLA#11 at 4894c4ad. The inputs match CLA#11's interface.

As in blastdns#139, the PyPI job only downloads, attests and publishes, and crates.io goes through CLA's crates.yml with the OpenSSL setup script. The old publishers are removed: python-tests.yml (MATURIN_PYPI_TOKEN upload at line 270) is deleted, and rust-publish (cargo publish --token at line 80) is gone from rust-tests.yml. Deleting release.toml (cargo-release, which pushed v{{version}} tags) is right too. rust-tests still reports under its live name.

High

H1. The release App key can't come from the release environment (C1).
publish.yml:73 and :79 pass vars.RELEASE_APP_ID / secrets.RELEASE_APP_KEY from a uses: job, which can't hold environment:. Neither exists, and the spec's names are BLS_RELEASE_APP_ID / BLS_RELEASE_PRIVATE_KEY, held in release.
Requested: once CLA#11 reads them inside its environment job, drop release-app-id and the secrets: block.

H2. The Python required checks on stable will never report.
Ruleset 14658129 requires python-tests (3.11)–(3.15) and rust-tests. This head reports rust-tests but python / test (3.10)…(3.15) for Python.
Requested: change the ruleset to the new Python contexts before the promotion (coordinated with .github-config#1/#2), or keep the live names.

Medium

M1. rust-tests.yml runs in the release workflow unpinned and with default permissions.
rust-tests.yml:18, :21, :27 and :36 use actions/checkout@v7, setup-rust-toolchain@v2, actions/cache@v6 and setup-python@v7, and the file has no permissions: block. publish.yml:23–26 now calls it in the release run. The description's "grep … @[0-9a-f]{40} → (none)" doesn't hold at this head.
Requested: pin all four to full SHAs and add permissions: contents: read. Optionally, add persist-credentials: false on the checkout.

M2. crates.io publishes in parallel with PyPI. crates (publish.yml:37–43) needs only wheels.
Requested: crates: needs: [check, pypi].

M3. Version state (C3).
dev is 0.10.0 and stable is 0.10.1, both already on PyPI and crates.io with no v* release.
Requested: the promotion should carry 0.10.2 or later.

M4. Stale description and pin (C2, C4, C5).
The description says human-pushed tags and links release.sh, and its admin step registers PyPI against pypi.
Requested: change the environment to release, re-pin to the CLA#11 merge commit (tests.yml:18 and cla.yml are on 40c6e18), and hold the promotion until release is protected.

Low

  • L1. Duplicate test runs. tests.yml and rust-tests.yml still trigger on push: stable, with groups tests-${{ github.ref }} and rust-tests-${{ github.ref }}, the same groups as the calls from publish.yml (C7).
  • L2. cla.yml inherits every secret under pull_request_target. Pass APP_ID/APP_PRIVATE_KEY by name (C6).
  • L3. Policy link. The README links .github-config/blob/stable/docs/policy/publishing.md, which is in a private repo and isn't merged yet (.github-config#17). The link won't resolve for public readers.

Verdict

Request changes: H1 (depends on CLA#11), H2 before the promotion, and M1–M4.

Next: we're making these changes ourselves in a follow-up PR, which will be linked here when it opens. This review is the record of what that PR addresses.

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Publish workflow won’t start: the crates job doesn’t grant id-token: write

publish.yml sets permissions: contents: read for the whole workflow. The crates job calls the shared crates.yml without a permissions: block of its own, so it only gets contents: read.
The shared crates.yml (CLA at 4894c4a) needs id-token: write on its job, because crates.io trusted publishing uses an OIDC login (crates-io-auth-action).
A called workflow can’t get more permissions than its caller gives it. GitHub checks this before any job runs, so every push to stable gives the whole Publish run a startup failure. The error is “nested job ‘crates’ is requesting ‘id-token: write’, but is only allowed ‘id-token: none’”. Nothing publishes, and even the check job never runs.
The PR checks can’t catch this, because publish.yml only runs on a push to stable.

Fix:

yaml
crates:
needs: [check, wheels]
if: needs.check.outputs.release == 'true'
permissions:
contents: read
id-token: write
uses: blacklanternsecurity/CLA@4894c4a...

@singlerider singlerider added the github_actions Pull requests that update GitHub Actions code label Oct 22, 2026
# Conflicts:
#	.github/workflows/python-tests.yml
#	pyproject.toml
- publish: publish crates.io after PyPI, granting the id-token the
  shared crates.yml needs, so the workflow no longer fails at startup;
  pass no App credentials
- rust-tests: pin every action to a SHA, declare contents: read, and
  keep the token out of .git/config
- tests, rust-tests: drop the stable push trigger and separate the
  concurrency groups from the publish calls; keep reporting stable's
  python-tests (3.11) to (3.15) contexts until the ruleset moves
- cla: pass the CLA App credentials by name instead of secrets: inherit
- stable-guard: refuse a pull request into stable whose version or
  dependencies cannot release
- README: link the public org release standard
- bump to 0.10.2: 0.10.1 is on PyPI and crates.io with no v0.10.1 tag
- re-pin every CLA reference to CLA#11
@singlerider singlerider changed the title ci: run tests and tag-only publish through the shared workflows ci: run tests and release on merge to stable through the shared workflows Oct 23, 2026

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared workflows are pinned to a CLA PR that hasn’t merged. Every uses: line (in cla.yml, dependabot-auto-merge.yml, publish.yml, tests.yml and stable-guard.yml) points at 07e14bc. That commit is on CLA#11, which is still Open. The PR description itself says to re-pin to CLA#11’s merge commit before this merges. So the order is: merge CLA#11, re-pin all five files, then merge this.

@pjhartlieb pjhartlieb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@singlerider, re-reviewed at 7968f0a2.

From the 2026-10-21 review

Status at 7968f0a2
H1 App credentials Fixed. No App lines in callers; publish.yml:68-78
H2 required checks on stable Fixed. tests.yml:21-33 reports python-tests (3.11)–(3.15); rust-tests.yml:15 keeps rust-tests
M1 rust-tests.yml pins and scope Fixed. rust-tests.yml:7-8 contents: read, :18-38 SHA pins, :20 persist-credentials: false, PR and workflow_call only
M2 crates after PyPI Fixed. publish.yml:38, with id-token: write at :40-42
M3 version state Fixed. Cargo.toml 0.10.2: no tag, not on PyPI or crates.io. It releases only when promoted to stable
M4 description, pin Description fixed. Pin: see 1 below
L1 duplicate runs Fixed. No test workflow triggers on push: stable
L2 cla.yml inherit Fixed. cla.yml:16-18
L3 policy link Fixed. README.md:620 links the public .github standard, which resolves anonymously

The wheel targets (publish.yml:33) cover all 10 platforms that 0.10.1 shipped. PYO3_USE_ABI3_FORWARD_COMPATIBILITY reaches both the tests and the wheels.

Still open

  1. Pin. Every CLA ref is CLA#11's unmerged head 07e14bcb. Re-pin to the CLA merge commit before this merges.
  2. dependabot-auto-merge.yml is still on dev (R11). Delete it in this PR.
  3. publish.yml:7-8 is contents: read. Make it permissions: {} and grant contents: read on check, tests, rust-tests and wheels.
  4. Release docs: a release prompts for release approval four times: release-check's credentials job, pypi, crates and the release.

Reproduce

TMPDIR=<scratch dir> bash verify-blasthttp-120.sh 7968f0a22ed4ec7a7c90132446c55ead84483d0d
# expect: SUMMARY repo=blasthttp pr=120 sha=7968f0a2... pass=24 fail=2   (P13, P14 = items 3, 2)
bash verify-blasthttp-120.sh 349e58d5af6900b67592f3d171676bc50e2ac1c7
# negative control, the head reviewed 2026-10-21: pass=12 fail=14 (P02 P04 P05 P06 P07 P08 P11 P12 P13 P14 P15.stable P16 H01 H04)

It needs git, an authenticated gh (GET calls only), uv and curl. It downloads actionlint 1.7.12 and checks it against the release checksums.

verify-blasthttp-120.sh
#!/usr/bin/env bash
# verify-blasthttp-120.sh <sha>
#
# Read-only re-check of blacklanternsecurity/blasthttp#120 (publish on merge to stable)
# against the 2026-10-21 review findings and go-live plan item A4 (+ R6, R11).
# Clean blobless clone of the repo at <sha>; GitHub reads are `gh api --method GET`
# only; PyPI and crates.io reads are anonymous GETs. Nothing is written anywhere
# except a temp dir under $TMPDIR (removed on exit).
#
# Prints PASS/FAIL/INFO per check and a final SUMMARY line; exits 1 on any FAIL.
# Needs: git, gh (authenticated), uv, curl, shasum or sha256sum.
# Env: CLA_SHA (expected CLA pin, default the CLA#11 head reviewed 2026-10-23).
set -uo pipefail

OWNER=blacklanternsecurity
REPO=blasthttp
PR=120
SHA=${1:?usage: $0 <commit-sha>}
CLA_SHA=${CLA_SHA:-07e14bcbba5e737e1b29bd9e3f9128fbf2483be9}
ACTIONLINT_VERSION=1.7.12

WORK=$(mktemp -d "${TMPDIR:-$PWD}/verify-$REPO-$PR.XXXXXX")
trap 'rm -rf "$WORK"' EXIT

echo "verify $OWNER/$REPO#$PR at $SHA (CLA_SHA=$CLA_SHA) $(date -u +%Y-%m-%dT%H:%M:%SZ)"

git clone --quiet --filter=blob:none --no-checkout "https://github.com/$OWNER/$REPO.git" "$WORK/repo" || { echo "FAIL clone"; echo "SUMMARY repo=$REPO sha=$SHA pass=0 fail=1"; exit 1; }
git -C "$WORK/repo" fetch --quiet origin "$SHA" 2>/dev/null \
  || git -C "$WORK/repo" fetch --quiet origin "+refs/pull/$PR/head:refs/remotes/origin/pr"
git -C "$WORK/repo" -c advice.detachedHead=false checkout --quiet "$SHA" || { echo "FAIL checkout $SHA"; echo "SUMMARY repo=$REPO sha=$SHA pass=0 fail=1"; exit 1; }

# actionlint, downloaded from the release and verified against the release's checksums file
case "$(uname -s)_$(uname -m)" in
  Darwin_arm64) PLAT=darwin_arm64 ;; Darwin_x86_64) PLAT=darwin_amd64 ;;
  Linux_x86_64) PLAT=linux_amd64 ;; Linux_aarch64|Linux_arm64) PLAT=linux_arm64 ;;
  *) PLAT="" ;;
esac
ACTIONLINT=""
if [ -n "$PLAT" ]; then
  base="https://github.com/rhysd/actionlint/releases/download/v$ACTIONLINT_VERSION"
  tgz="actionlint_${ACTIONLINT_VERSION}_${PLAT}.tar.gz"
  sums="actionlint_${ACTIONLINT_VERSION}_checksums.txt"
  mkdir -p "$WORK/al"
  if curl -sSfL -o "$WORK/al/$tgz" "$base/$tgz" && curl -sSfL -o "$WORK/al/$sums" "$base/$sums"; then
    want=$(grep " $tgz\$" "$WORK/al/$sums" | cut -d' ' -f1)
    got=$( (shasum -a 256 "$WORK/al/$tgz" 2>/dev/null || sha256sum "$WORK/al/$tgz") | cut -d' ' -f1)
    if [ -n "$want" ] && [ "$want" = "$got" ]; then
      tar -xzf "$WORK/al/$tgz" -C "$WORK/al" actionlint && ACTIONLINT="$WORK/al/actionlint"
      echo "INFO actionlint $ACTIONLINT_VERSION $PLAT sha256 $got matches the release checksums file"
    else
      echo "INFO actionlint sha256 mismatch (want=$want got=$got); not used"
    fi
  fi
fi

cat > "$WORK/check.py" <<'PY'
import base64, json, os, re, subprocess, sys, tomllib, urllib.request, urllib.error
import yaml

owner, repo, pr, sha, cla_sha, root, actionlint = sys.argv[1:8]
wfdir = os.path.join(root, ".github", "workflows")
results = []

def out(status, cid, desc, detail=""):
    results.append(status)
    print(f"{status} {cid} {desc}" + (f" -- {detail}" if detail else ""))

def check(cid, desc, ok, detail=""):
    out("PASS" if ok else "FAIL", cid, desc, detail)

def info(cid, desc, detail=""):
    print(f"INFO {cid} {desc}" + (f" -- {detail}" if detail else ""))

def gh(path):
    cmd = ["gh", "api", "--method", "GET", path]
    p = subprocess.run(cmd, capture_output=True, text=True)
    return p.returncode, p.stdout

def http_status(url):
    req = urllib.request.Request(url, headers={"User-Agent": f"{owner}-{repo}-pr-review-readonly"})
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            return r.status, r.read()
    except urllib.error.HTTPError as e:
        return e.code, b""

def load(path):
    with open(path) as f:
        return yaml.safe_load(f)

def triggers(wf):
    on = wf.get("on", wf.get(True))
    if isinstance(on, str):
        return {on: None}
    if isinstance(on, list):
        return {k: None for k in on}
    return on or {}

files = sorted(f for f in os.listdir(wfdir) if f.endswith((".yml", ".yaml"))) if os.path.isdir(wfdir) else []
wfs = {f: load(os.path.join(wfdir, f)) for f in files}
texts = {f: open(os.path.join(wfdir, f)).read() for f in files}
pub = wfs.get("publish.yml")
jobs = (pub or {}).get("jobs", {}) or {}

def needs(job):
    n = (jobs.get(job) or {}).get("needs", [])
    return [n] if isinstance(n, str) else list(n)

def job_using(fragment):
    return [j for j, d in jobs.items() if fragment in str((d or {}).get("uses", ""))]

# P01 trigger
t = triggers(pub or {})
push = t.get("push") or {}
check("P01", "publish.yml triggers only on push to stable",
      pub is not None and set(t) == {"push"} and push.get("branches") == ["stable"] and not push.get("tags"),
      f"on={json.dumps(t, default=str)}")

# P02 job order: tests -> wheels -> pypi -> crates -> release
wheels = job_using("maturin-wheels.yml") or [j for j in jobs if j == "wheels"]
crates = job_using("CLA/.github/workflows/crates.yml") or [j for j in jobs if j == "crates"]
rel = job_using("CLA/.github/workflows/publish.yml")
pypi = [j for j, d in jobs.items() if "pypa/gh-action-pypi-publish" in json.dumps(d) or "\"command\": \"upload\"" in json.dumps(d)]
local_tests = [j for j, d in jobs.items() if str((d or {}).get("uses", "")).startswith("./")]
order = []
if wheels and pypi and crates and rel:
    w, p, c, r = wheels[0], pypi[0], crates[0], rel[0]
    order = [set(local_tests) <= set(needs(w)) and bool(local_tests), w in needs(p), p in needs(c), {c, p} <= set(needs(r))]
check("P02", "job order tests -> wheels -> pypi -> crates -> release",
      bool(order) and all(order),
      f"tests={local_tests} wheels.needs={needs(wheels[0]) if wheels else None} pypi.needs={needs(pypi[0]) if pypi else None} "
      f"crates.needs={needs(crates[0]) if crates else None} release.needs={needs(rel[0]) if rel else None}")

# P03 pypi job
pj = jobs.get(pypi[0]) if pypi else {}
steps = json.dumps((pj or {}).get("steps", []))
check("P03", "pypi job: environment release, id-token: write, skip-existing",
      bool(pj) and pj.get("environment") == "release" and (pj.get("permissions") or {}).get("id-token") == "write"
      and re.search(r'"skip-existing": (true|"true")|--skip-existing', steps) is not None,
      f"environment={pj.get('environment') if pj else None} permissions={pj.get('permissions') if pj else None}")

# P04 crates via CLA crates.yml with id-token
cj = jobs.get(crates[0]) if crates else {}
check("P04", "crates job calls CLA crates.yml and grants id-token: write",
      bool(cj) and "CLA/.github/workflows/crates.yml" in str(cj.get("uses", "")) and (cj.get("permissions") or {}).get("id-token") == "write",
      f"permissions={cj.get('permissions') if cj else None}")

alltext = "\n".join(f"{f}:{i+1}:{l}" for f, s in texts.items() for i, l in enumerate(s.splitlines()))

def hits(rx):
    return [l for l in alltext.splitlines() if re.search(rx, l.split(":", 2)[2])]

# P05 no App credential lines in callers
h = hits(r"release-app-|RELEASE_APP_|BLS_RELEASE_")
check("P05", "no release App credential lines in callers (CLA reads them in `release`)", not h, "; ".join(h[:4]))

# P06 no secrets: inherit
h = hits(r"secrets:\s*inherit")
check("P06", "no `secrets: inherit` in any workflow", not h, "; ".join(h[:4]))

# P07 SHA pins
bad = []
for l in alltext.splitlines():
    m = re.search(r"^\s*(?:-\s+)?uses:\s*([^\s#]+)", l.split(":", 2)[2])
    if m and not m.group(1).startswith("./") and not re.search(r"@[0-9a-f]{40}$", m.group(1)):
        bad.append(l.strip())
check("P07", "every `uses:` is local or pinned to a full SHA", not bad, "; ".join(bad[:4]))

# P08 / P09 CLA pins
cla_refs = sorted({m.group(1, 2) for m in re.finditer(r"blacklanternsecurity/CLA/(\.github/workflows/[\w.-]+)@([0-9a-f]{7,40})", alltext)})
pins = sorted({s for _, s in cla_refs})
check("P08", f"every CLA reference pinned to CLA_SHA {cla_sha[:8]}", pins == [cla_sha], f"pins={[p[:8] for p in pins]}")
missing = []
for path, s in cla_refs:
    rc, _ = gh(f"repos/blacklanternsecurity/CLA/contents/{path}?ref={s}")
    if rc != 0:
        missing.append(f"{path}@{s[:8]}")
check("P09", "every referenced CLA workflow exists at its pin", bool(cla_refs) and not missing,
      f"{len(cla_refs)} refs" + (f"; missing {missing}" if missing else ""))

# P10 old publish paths
h = hits(r"PYPI_TOKEN|PYPI_API_TOKEN|MATURIN_PYPI_TOKEN|secrets\.CARGO_REGISTRY_TOKEN|cargo publish .*--token|maturin publish|twine upload|gh release create|git push .*--tags|git tag ")
extra = [f for f in ("release.toml", "release.sh") if os.path.exists(os.path.join(root, f))]
check("P10", "no old publish path in any workflow (stored tokens, cargo publish --token, maturin publish, tag pushes)", not h and not extra, "; ".join(h[:4] + extra))

# P11 no other workflow runs on push to stable (duplicate test runs, C7)
dup = []
for f, wf in wfs.items():
    if f == "publish.yml":
        continue
    tt = triggers(wf)
    if "push" in tt:
        b = (tt.get("push") or {}).get("branches")
        if b is None or "stable" in b or any("*" in x for x in b):
            dup.append(f)
check("P11", "no workflow besides publish.yml triggers on push to stable", not dup, f"{dup}")

# P12 explicit top-level permissions (R6)
noperm = [f for f, wf in wfs.items() if "permissions" not in wf]
check("P12", "every workflow declares top-level permissions (R6)", not noperm, f"{noperm}")

# P13 permissions: {} on publish.yml (A4)
check("P13", "publish.yml top-level `permissions: {}` (A4)", pub is not None and pub.get("permissions") == {},
      f"permissions={pub.get('permissions') if pub else None}")

# P14 dependabot-auto-merge removed from dev (R11)
dam = [f for f in files if "dependabot-auto-merge" in f]
check("P14", "dependabot-auto-merge workflow removed (R11)", not dam, f"{dam}")

# P15 live required checks reported at this SHA
p = subprocess.run(["gh", "api", "--method", "GET", "--paginate", f"repos/{owner}/{repo}/commits/{sha}/check-runs?per_page=100",
                    "--jq", ".check_runs[] | {name, status, conclusion, app: {id: .app.id}}"], capture_output=True, text=True)
runs = [json.loads(l) for l in p.stdout.splitlines() if l.strip()]
reported = {(r["name"], (r.get("app") or {}).get("id")) for r in runs}
for branch in ("dev", "stable"):
    rc, body = gh(f"repos/{owner}/{repo}/rules/branches/{branch}")
    req = []
    for rule in json.loads(body or "[]"):
        if rule.get("type") == "required_status_checks":
            p = rule["parameters"]
            req += [(c["context"], c.get("integration_id"), rule.get("ruleset_id"), p.get("strict_required_status_checks_policy")) for c in p["required_status_checks"]]
    miss = [c for c, i, _, _ in req if not any(n == c and (i is None or a == i) for n, a in reported)]
    strict = sorted({s for *_, s in req})
    check(f"P15.{branch}", f"every live `{branch}` required check is reported at this SHA",
          not miss, f"required={len(req)} strict={strict} missing={miss}")

# P16 version not already published (C3)
cargo = tomllib.load(open(os.path.join(root, "Cargo.toml"), "rb"))
pyproj = tomllib.load(open(os.path.join(root, "pyproject.toml"), "rb"))
ver = cargo["package"]["version"]
crate = cargo["package"]["name"]
pyname = pyproj.get("project", {}).get("name")
pyver = pyproj.get("project", {}).get("version")
rc_tag, _ = gh(f"repos/{owner}/{repo}/git/ref/tags/v{ver}")
s_pypi, _ = http_status(f"https://pypi.org/pypi/{pyname}/{ver}/json")
s_crates, _ = http_status(f"https://crates.io/api/v1/crates/{crate}/{ver}")
check("P16", f"manifest version {ver} is unreleased (no v{ver} tag, not on PyPI or crates.io)",
      rc_tag != 0 and s_pypi == 404 and s_crates == 404 and (pyver in (None, ver)),
      f"tag_exists={rc_tag == 0} pypi={s_pypi} crates={s_crates} pyproject.version={pyver}")

# P17 actionlint
if actionlint:
    p = subprocess.run([actionlint, "-shellcheck=", "-pyflakes=", "-no-color"], cwd=root, capture_output=True, text=True)
    check("P17", "actionlint 1.7.12 clean (shellcheck/pyflakes integration off)", p.returncode == 0, (p.stdout + p.stderr).strip().replace("\n", " | ")[:600])
else:
    out("FAIL", "P17", "actionlint 1.7.12 clean", "actionlint unavailable or checksum mismatch")

# P18 CI at this SHA
bad = sorted({r["name"] for r in runs if r.get("status") == "completed" and r.get("conclusion") not in ("success", "skipped", "neutral")})
pend = sorted({r["name"] for r in runs if r.get("status") != "completed"})
check("P18", "every check run at this SHA concluded success/skipped/neutral", bool(runs) and not bad and not pend,
      f"runs={len(runs)} failing={bad} pending={pend}")

# W: wheel matrix through CLA maturin-wheels.yml, compared with the latest PyPI release
wj = jobs.get(wheels[0]) if wheels else None
via_cla = bool(wj) and "CLA/.github/workflows/maturin-wheels.yml" in str(wj.get("uses", ""))
check("W00", "wheels built by CLA maturin-wheels.yml", via_cla, f"wheels job={wheels[0] if wheels else None}")
s, raw = http_status(f"https://pypi.org/pypi/{pyname}/json")
latest = json.loads(raw) if s == 200 else {}
wheels_pypi = [u["filename"] for u in latest.get("urls", []) if u["filename"].endswith(".whl")]
arch = {"x86_64": "x86_64", "i686": "x86", "aarch64": "aarch64", "armv7l": "armv7", "s390x": "s390x", "ppc64le": "ppc64le",
        "amd64": "x64", "win32": "x86", "arm64": "aarch64"}
def fam(plat):
    a = next((v for k, v in arch.items() if plat.endswith(k)), plat)
    for prefix, family in (("musllinux", "musllinux"), ("manylinux", "linux"), ("linux", "linux"), ("macosx", "macos"), ("win", "windows")):
        if plat.startswith(prefix):
            return family, a
    return "other", plat
shipped = set()
interp = set()
for fn in wheels_pypi:
    parts = fn[:-4].split("-")
    for plat in parts[4].split("."):
        f, a = fam(plat)
        shipped.add((f, a))
    interp.add(f"{parts[2]}-{parts[3]}")
if via_cla:
    m = re.search(r"maturin-wheels\.yml@([0-9a-f]{40})", str(wj["uses"]))
    rc, src = gh(f"repos/blacklanternsecurity/CLA/contents/.github/workflows/maturin-wheels.yml?ref={m.group(1)}") if m else (1, "")
    clawf = yaml.safe_load(base64.b64decode(json.loads(src)["content"]).decode()) if rc == 0 else {}
    default = clawf.get("on", clawf.get(True))["workflow_call"]["inputs"]["targets"]["default"]
    targets = json.loads((wj.get("with") or {}).get("targets") or default)
    built = {(f, a) for f, al in targets.items() for a in al}
    gap = sorted(shipped - built)
    check("W01", f"platforms built cover every platform of {pyname} {latest.get('info', {}).get('version')} on PyPI",
          not gap, f"shipped={len(shipped)} built={len(built)} not_built={gap}")
    rc2, src2 = gh(f"repos/blacklanternsecurity/CLA/contents/scripts/manifest.py?ref={m.group(1)}")
    mp = base64.b64decode(json.loads(src2)["content"]).decode() if rc2 == 0 else ""
    cpython_default_only = 'build["implementation"] != "cpython"' in mp and 'build["variant"] != "default"' in mp
    dropped = sorted(i for i in interp if not re.fullmatch(r"cp3(\d+)-cp3\1", i)) if cpython_default_only else []
    check("W02", "every interpreter ABI on the latest PyPI release is still built (CLA builds released CPython, default variant, only)",
          not dropped, f"pypi_abis={sorted(interp)} not_built={dropped}")
else:
    out("FAIL", "W01", "platform coverage", "depends on W00")
    out("FAIL", "W02", "interpreter coverage", "depends on W00")

# repo-specific checks
extra = os.environ.get("EXTRA_CHECKS", "")
if "blasthttp" in extra:
    rt = wfs.get("rust-tests.yml") or {}
    rtt = texts.get("rust-tests.yml", "")
    unp = [u for u in re.findall(r"(?m)^\s*(?:-\s+)?uses:\s*([^\s#]+)", rtt) if not re.search(r"@[0-9a-f]{40}$", u)]
    tt = triggers(rt)
    check("H01", "rust-tests.yml: SHA-pinned, contents: read, no push trigger, persist-credentials: false",
          bool(rt) and not unp and rt.get("permissions") == {"contents": "read"} and "push" not in tt and "persist-credentials: false" in rtt,
          f"unpinned={unp} permissions={rt.get('permissions')} triggers={list(tt)}")
    ok = "rust-tests" in jobs and "rust-tests" in needs(wheels[0] if wheels else "")
    check("H02", "publish.yml runs rust-tests.yml before the wheels", ok, f"wheels.needs={needs(wheels[0]) if wheels else None}")
    env_tests = "PYO3_USE_ABI3_FORWARD_COMPATIBILITY" in texts.get("tests.yml", "")
    env_wheels = "PYO3_USE_ABI3_FORWARD_COMPATIBILITY" in json.dumps(wj or {})
    check("H03", "PYO3_USE_ABI3_FORWARD_COMPATIBILITY passed to the shared tests and wheels (en0f)", env_tests and env_wheels,
          f"tests={env_tests} wheels={env_wheels}")
    readme = open(os.path.join(root, "README.md")).read()
    links = sorted(set(re.findall(r"https://github\.com/blacklanternsecurity/\.github[\w.-]*/blob/[^\s)]+", readme)))
    st = {l: http_status(l)[0] for l in links}
    check("H04", "README release-policy link resolves for an anonymous reader", bool(links) and all(v == 200 for v in st.values()), f"{st}")

n_pass, n_fail = results.count("PASS"), results.count("FAIL")
print(f"SUMMARY repo={repo} pr={pr} sha={sha} pass={n_pass} fail={n_fail}")
sys.exit(1 if n_fail else 0)
PY

echo "INFO env release environment exists: $(gh api --method GET "repos/$OWNER/$REPO/environments" --jq '[.environments[].name] | index("release") != null' 2>/dev/null || echo unknown)"
EXTRA_CHECKS="blasthttp" uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.2 python -I "$WORK/check.py" \
  "$OWNER" "$REPO" "$PR" "$SHA" "$CLA_SHA" "$WORK/repo" "$ACTIONLINT"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants