Skip to content

Commit 09b94e2

Browse files
committed
CI: harden build-wheel.yml for the eMudhra/emSign CA incident (12.9.x backport)
Backport the CI hardening that main received for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557), so that wheels released from this branch, and the cuda-core wheels on main that are built against this branch's cuda-bindings wheels, are built with patched tooling and verified downloads: - upgrade-ca-certificates: upgrade ca-certificates on GitHub-hosted Linux and distrust the compromised intermediates on GitHub-hosted Windows, as the first step of build-wheel.yml (#3270). - setup-hardened-python: install interpreters with a pinned uv and fail on unpatched ones, replacing actions/setup-python + harden-pip in build-wheel.yml and test-wheel-{linux,windows}.yml (#3275). This branch still builds and tests Python 3.10, which has no fixed release; for series older than 3.11 the action warns and installs pip from the sha256-pinned 26.2.2 wheel instead of failing. cibuildwheel now runs with the hardened host Python (`python-path`), and the Python include path is derived from sysconfig because uv interpreters are not at Python3_ROOT_DIR. - cibuildwheel: install the sha256-pinned pip 26.2.2 in CIBW_BEFORE_BUILD_LINUX and CIBW_BEFORE_BUILD_WINDOWS before anything else is downloaded (Linux had no before-build step; Windows ran an unpinned `pip install delvewheel`). - fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and every component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch (ci/tools/fetch_ctk_redistrib.py, with tests). Also stop passing `-k` to curl, which disabled TLS verification for the archive downloads. - install_gpu_driver.ps1: verify the fixed Windows driver installer against a pinned sha256 before running it as admin (#3249 for this branch's single installer). This branch has no sccache and no Linux driver install script, so the install-sccache action and the Linux part of #3249 do not apply. Not backported: the runner trust probe and the "Enable Git symlinks" steps. Refs #3195
1 parent 915a2c2 commit 09b94e2

12 files changed

Lines changed: 1348 additions & 72 deletions

File tree

‎.github/actions/fetch_ctk/action.yml‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# SPDX-License-Identifier: Apache-2.0
44

@@ -76,19 +76,22 @@ runs:
7676
mkdir $CACHE_TMP_DIR
7777
7878
# The binary archives (redist) are guaranteed to be updated as part of the release posting.
79+
# Use the runtime workspace mount so this also works inside container jobs.
80+
CTK_REDIST_TOOL="${GITHUB_WORKSPACE}/ci/tools/fetch_ctk_redistrib.py"
7981
CTK_BASE_URL="https://developer.download.nvidia.com/compute/cuda/redist/"
8082
CTK_JSON_URL="$CTK_BASE_URL/redistrib_${{ inputs.cuda-version }}.json"
83+
CTK_JSON_FILE="$CACHE_TMP_DIR/redistrib.json"
84+
curl -LSs "$CTK_JSON_URL" -o "$CTK_JSON_FILE"
85+
# Pin the metadata JSON: the per-archive sha256 checks below are only
86+
# trustworthy if the JSON that carries those hashes is itself trusted.
87+
python "$CTK_REDIST_TOOL" verify-metadata \
88+
--cuda-version "${{ inputs.cuda-version }}" \
89+
--metadata-path "$CTK_JSON_FILE"
8190
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
82-
if [[ "${{ inputs.host-platform }}" == "linux-64" ]]; then
83-
CTK_SUBDIR="linux-x86_64"
84-
elif [[ "${{ inputs.host-platform }}" == "linux-aarch64" ]]; then
85-
CTK_SUBDIR="linux-sbsa"
86-
fi
8791
function extract() {
8892
tar -xvf $1 -C $CACHE_TMP_DIR --strip-components=1
8993
}
9094
elif [[ "${{ inputs.host-platform }}" == "win-64" ]]; then
91-
CTK_SUBDIR="windows-x86_64"
9295
function extract() {
9396
_TEMP_DIR_=$(mktemp -d)
9497
unzip $1 -d $_TEMP_DIR_
@@ -101,14 +104,23 @@ runs:
101104
function populate_cuda_path() {
102105
# take the component name as a argument
103106
function download() {
104-
curl -kLSs $1 -o $2
107+
curl -LSs $1 -o $2
105108
}
106109
CTK_COMPONENT=$1
107-
CTK_COMPONENT_REL_PATH="$(curl -s $CTK_JSON_URL |
108-
python -c "import sys, json; print(json.load(sys.stdin)['${CTK_COMPONENT}']['${CTK_SUBDIR}']['relative_path'])")"
110+
CTK_COMPONENT_REL_PATH="$(python "$CTK_REDIST_TOOL" component-relative-path \
111+
--host-platform "${{ inputs.host-platform }}" \
112+
--component "$CTK_COMPONENT" \
113+
--metadata-path "$CTK_JSON_FILE")"
109114
CTK_COMPONENT_URL="${CTK_BASE_URL}/${CTK_COMPONENT_REL_PATH}"
110115
CTK_COMPONENT_COMPONENT_FILENAME="$(basename $CTK_COMPONENT_REL_PATH)"
111116
download $CTK_COMPONENT_URL $CTK_COMPONENT_COMPONENT_FILENAME
117+
# Verify the downloaded archive against the sha256 in the (pinned)
118+
# metadata before extracting; fail closed on any mismatch.
119+
CTK_COMPONENT_SHA256="$(python "$CTK_REDIST_TOOL" component-sha256 \
120+
--host-platform "${{ inputs.host-platform }}" \
121+
--component "$CTK_COMPONENT" \
122+
--metadata-path "$CTK_JSON_FILE")"
123+
echo "${CTK_COMPONENT_SHA256} ${CTK_COMPONENT_COMPONENT_FILENAME}" | sha256sum --check --strict -
112124
extract $CTK_COMPONENT_COMPONENT_FILENAME
113125
rm $CTK_COMPONENT_COMPONENT_FILENAME
114126
}

‎.github/actions/harden-pip/action.yml‎

Lines changed: 0 additions & 28 deletions
This file was deleted.
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
name: Set up hardened Python
6+
description: >-
7+
Install CPython with a pinned uv, create a venv from it, and put the venv
8+
first on PATH. setup-uv verifies the uv binary and uv verifies the
9+
python-build-standalone archive against embedded checksums, so the
10+
interpreter's integrity does not depend on TLS. The job then fails unless
11+
the interpreter's ssl module rejects the compromised eMudhra/emSign CA keys
12+
(CVE-2026-115557; first fixed releases: 3.11.18, 3.12.16, 3.13.18, 3.14.10,
13+
3.15.2), and pip is updated through that ssl module. Unlike setup-python,
14+
this provides patched Windows builds of maintenance series, for which
15+
python.org publishes no installers. Series that have no fixed release
16+
(3.10 and older, which this branch still builds and tests) are installed
17+
with a warning instead, and their pip is replaced by a sha256-pinned
18+
wheel, so that the pip update does not depend on TLS. See
19+
https://github.com/NVIDIA/cuda-python/issues/3195 and
20+
https://discuss.python.org/t/109678.
21+
22+
inputs:
23+
python-version:
24+
description: >-
25+
uv Python request, e.g. "3.12" or "3.14t" (free-threaded). The newest
26+
patch known to the pinned uv is installed. Series with a fixed release
27+
listed in the interpreter check (3.11-3.15) must carry it. Series
28+
without a fixed release (3.10 and older) are accepted with a warning,
29+
and pip is installed from a sha256-pinned wheel.
30+
required: true
31+
venv:
32+
description: Directory for the venv. Defaults to a new directory under RUNNER_TEMP.
33+
default: ""
34+
35+
outputs:
36+
python-path:
37+
description: Absolute path of the venv's interpreter.
38+
value: ${{ steps.venv.outputs.python-path }}
39+
python-version:
40+
description: Full version of the interpreter, e.g. "3.14.10".
41+
value: ${{ steps.venv.outputs.python-version }}
42+
43+
runs:
44+
using: composite
45+
steps:
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
48+
with:
49+
# The newest uv whose checksum ships inside the pinned setup-uv, so
50+
# the binary is verified against the SHA-pinned action itself. The uv
51+
# release determines the newest patch of each CPython series, so
52+
# bumping the setup-uv SHA also moves the CPython patch versions.
53+
version: "latest-known"
54+
enable-cache: false
55+
# Some callers (e.g. release verification) have no checkout.
56+
ignore-empty-workdir: true
57+
58+
- name: Install Python and create venv
59+
id: venv
60+
shell: bash --noprofile --norc -euo pipefail {0}
61+
env:
62+
PY_REQUEST: ${{ inputs.python-version }}
63+
VENV_DIR: ${{ inputs.venv }}
64+
# A per-job install directory ensures the newest patch known to the
65+
# pinned uv is used, never an older interpreter left on a persistent
66+
# self-hosted runner.
67+
UV_PYTHON_INSTALL_DIR: ${{ runner.temp }}/setup-hardened-python/interpreters
68+
UV_NO_CONFIG: "1"
69+
run: |
70+
uv python install --no-bin --no-registry "${PY_REQUEST}"
71+
base="$(uv python find --managed-python --no-project "${PY_REQUEST}")"
72+
venv="${VENV_DIR}"
73+
if [[ -z "${venv}" ]]; then
74+
# A fresh directory per call, so a second call in the same job
75+
# (e.g. build-wheel.yml) leaves the first venv intact.
76+
venv="$("${base}" -c 'import sys, tempfile; print(tempfile.mkdtemp(prefix="venv-", dir=sys.argv[1]))' "${RUNNER_TEMP}")"
77+
fi
78+
# The stdlib venv seeds pip from the interpreter's bundled wheel, so
79+
# nothing is downloaded before the ssl check below.
80+
"${base}" -m venv --clear "${venv}"
81+
if [[ "${RUNNER_OS}" == Windows ]]; then
82+
py="${venv}/Scripts/python.exe"
83+
else
84+
py="${venv}/bin/python"
85+
fi
86+
# Let the interpreter report native absolute paths (Windows included).
87+
python_path="$("${py}" -c 'import sys; print(sys.executable)')"
88+
"${py}" -c 'import os, sys; print(os.path.dirname(sys.executable))' >> "${GITHUB_PATH}"
89+
{
90+
echo "python-path=${python_path}"
91+
echo "python-version=$("${py}" -c 'import platform; print(platform.python_version())')"
92+
} >> "${GITHUB_OUTPUT}"
93+
94+
- name: Check Python interpreter is patched (CVE-2026-115557)
95+
id: check
96+
# Runs the venv's python through the PATH entry added above.
97+
shell: python {0}
98+
env:
99+
PYTHON: ${{ steps.venv.outputs.python-path }}
100+
run: |
101+
import os
102+
import platform
103+
import sys
104+
105+
if os.path.normcase(sys.executable) != os.path.normcase(os.environ["PYTHON"]):
106+
print(f"::error::PATH resolves python to {sys.executable}, not the venv interpreter {os.environ['PYTHON']}.")
107+
sys.exit(1)
108+
109+
FIXED = {(3, 11): 18, (3, 12): 16, (3, 13): 18, (3, 14): 10, (3, 15): 2}
110+
# Series that will never get a fixed release (end of life before the
111+
# fix). Their pip is installed from a pinned wheel instead.
112+
UNPATCHABLE_BELOW = (3, 11)
113+
ssl_patched = "true"
114+
v = sys.version_info
115+
series = (v.major, v.minor)
116+
name = f"{v.major}.{v.minor}"
117+
full = platform.python_version()
118+
print("sys.version:", sys.version)
119+
if series in FIXED:
120+
floor = f"{name}.{FIXED[series]}"
121+
if v.micro < FIXED[series]:
122+
print(f"::error::Python {full} is older than {floor}, the first {name} release whose ssl module rejects the compromised eMudhra CA keys (CVE-2026-115557).")
123+
sys.exit(1)
124+
print(f"Python {full}: ssl module carries the fix (first fixed: {floor}).")
125+
elif series < UNPATCHABLE_BELOW:
126+
ssl_patched = "false"
127+
print(f"::warning::Python {full}: no {name} release rejects the compromised eMudhra CA keys (CVE-2026-115557); pip will be installed from a sha256-pinned wheel instead.")
128+
else:
129+
# Fail closed: a newer series is only accepted once its fixed release is listed.
130+
print(f"::error::Python {full}: no {name} release is known to reject the compromised eMudhra CA keys (CVE-2026-115557).")
131+
sys.exit(1)
132+
133+
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as f:
134+
f.write(f"ssl-patched={ssl_patched}\n")
135+
136+
- name: Update pip
137+
shell: bash --noprofile --norc -xeuo pipefail {0}
138+
env:
139+
PYTHON: ${{ steps.venv.outputs.python-path }}
140+
SSL_PATCHED: ${{ steps.check.outputs.ssl-patched }}
141+
run: |
142+
if [[ "${SSL_PATCHED}" == "true" ]]; then
143+
"${PYTHON}" -m pip install -U pip
144+
else
145+
# The interpreter's ssl module still trusts the compromised CA keys,
146+
# so do not fetch pip through it. The sha256 fragment guarantees the
147+
# wheel's integrity regardless of the transport.
148+
"${PYTHON}" -m pip install \
149+
"pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
150+
fi
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Temporary stopgap for the eMudhra/emSign CA key compromise (CVE-2026-111943)
6+
# on GitHub-hosted runners until GitHub ships fixed images.
7+
#
8+
# On Linux: upgrades ca-certificates via apt so the compromised emSign ECC Root
9+
# CA - G3 anchor is removed from /etc/ssl/certs/ca-certificates.crt. Ubuntu's
10+
# bundle does not include AAA Certificate Services, so the Sectigo cross-signs
11+
# do not chain to a trusted root there.
12+
# On Windows: adds the eight compromised intermediates and their eight Sectigo
13+
# cross-signs (impacted-intermediates.json) to Cert:\LocalMachine\Disallowed so
14+
# schannel-based clients (git, gh, PowerShell/.NET) reject chains through them,
15+
# including chains to AAA Certificate Services, which Windows still trusts.
16+
#
17+
# This action is a no-op on self-hosted runners (runner.environment != 'github-hosted')
18+
# because self-hosted runner operators own and update their own host images.
19+
#
20+
# Removal criteria and implementation notes: security/emudhra-plan2.md
21+
# Tracked by: actions/runner-images#15018
22+
23+
name: Harden OS certificate trust (eMudhra/emSign CA stopgap)
24+
description: >
25+
Temporary per-job stopgap for CVE-2026-111943. Upgrades ca-certificates on
26+
GitHub-hosted Linux runners and adds the compromised eMudhra intermediates
27+
and cross-signs to the Disallowed store on GitHub-hosted Windows runners.
28+
No-op on self-hosted.
29+
30+
runs:
31+
using: composite
32+
steps:
33+
34+
# -------------------------------------------------------------------------
35+
# Windows path: add the compromised intermediates to LocalMachine\Disallowed.
36+
# Takes effect immediately for all schannel-based clients in subsequent steps
37+
# (git, gh, PowerShell/.NET, WinHTTP). Must precede actions/checkout.
38+
# -------------------------------------------------------------------------
39+
- name: Distrust compromised eMudhra intermediates (Windows, GitHub-hosted)
40+
if: ${{ runner.os == 'Windows' && runner.environment == 'github-hosted' }}
41+
shell: pwsh
42+
env:
43+
ACTION_PATH: ${{ github.action_path }}
44+
run: |
45+
$ErrorActionPreference = 'Stop'
46+
. (Join-Path $env:ACTION_PATH 'emudhra_intermediates.ps1')
47+
48+
$certs = @(Get-EmudhraIntermediates)
49+
try {
50+
# Adding is idempotent, so no early exit when the image is already fixed.
51+
$store = [System.Security.Cryptography.X509Certificates.X509Store]::new('Disallowed', 'LocalMachine')
52+
$store.Open('ReadWrite')
53+
try {
54+
foreach ($cert in $certs) { $store.Add($cert) }
55+
} finally {
56+
$store.Close()
57+
}
58+
59+
$notDistrusted = @($certs | Where-Object { -not (Test-ExplicitlyDistrusted $_) } | ForEach-Object { $_.EmudhraLabel })
60+
if ($notDistrusted.Count -gt 0) {
61+
Write-Error "Not explicitly distrusted after adding to LocalMachine\Disallowed: $($notDistrusted -join '; ')"
62+
exit 1
63+
}
64+
Write-Host "Distrusted $($certs.Count) compromised eMudhra intermediates and cross-signs."
65+
} finally {
66+
$certs | ForEach-Object { $_.Dispose() }
67+
}
68+
69+
# -------------------------------------------------------------------------
70+
# Linux path: upgrade ca-certificates via apt.
71+
# The Ubuntu archive GPG check does not depend on the trust store being
72+
# fixed, so this bootstrap is safe. Retries guard against transient mirror
73+
# failures. Must precede actions/checkout.
74+
# -------------------------------------------------------------------------
75+
- name: Upgrade ca-certificates (Linux, GitHub-hosted)
76+
if: ${{ runner.os == 'Linux' && runner.environment == 'github-hosted' }}
77+
shell: bash
78+
run: |
79+
set -euo pipefail
80+
EMSIGN_G3_FP="86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B"
81+
CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"
82+
VERSION_ID="$(. /etc/os-release && echo "${VERSION_ID}")"
83+
84+
# Fail closed if openssl is missing or the bundle cannot be parsed
85+
# (mirrors ci/tools/runner_trust_probe.sh).
86+
if ! command -v openssl >/dev/null 2>&1; then
87+
echo "::error::openssl not found; cannot verify CA bundle"
88+
exit 1
89+
fi
90+
if [[ ! -f "${CA_BUNDLE}" ]]; then
91+
echo "::error::CA bundle not found: ${CA_BUNDLE}"
92+
exit 1
93+
fi
94+
95+
fingerprints_in_bundle() {
96+
awk '/-----BEGIN CERTIFICATE-----/{cert=""} {cert=cert $0 "\n"}
97+
/-----END CERTIFICATE-----/{print cert | "openssl x509 -noout -fingerprint -sha256 2>/dev/null"; close("openssl x509 -noout -fingerprint -sha256 2>/dev/null")}' \
98+
"${CA_BUNDLE}"
99+
}
100+
101+
fps="$(fingerprints_in_bundle)"
102+
cert_count="$(grep -c . <<< "${fps}" || true)"
103+
if (( cert_count == 0 )); then
104+
echo "::error::No certificates parsed from ${CA_BUNDLE}; openssl may have failed"
105+
exit 1
106+
fi
107+
108+
if grep -qiF "${EMSIGN_G3_FP}" <<< "${fps}"; then
109+
before_state="not fixed"
110+
else
111+
before_state="already fixed"
112+
fi
113+
before_ver="$(dpkg-query -W -f='${Version}' ca-certificates 2>/dev/null || echo unknown)"
114+
echo "::notice::emSign ECC Root CA - G3 on ${VERSION_ID}: ${before_state} (ca-certificates ${before_ver})"
115+
116+
if [ "${before_state}" = "already fixed" ]; then
117+
exit 0
118+
fi
119+
120+
for attempt in 1 2 3; do
121+
if sudo apt-get update -q && sudo apt-get install --only-upgrade -y ca-certificates; then
122+
break
123+
fi
124+
if [ "${attempt}" -lt 3 ]; then
125+
echo "apt-get attempt ${attempt} failed; retrying in 10s..."
126+
sleep 10
127+
else
128+
echo "::error::apt-get upgrade failed after ${attempt} attempts"
129+
exit 1
130+
fi
131+
done
132+
133+
after_ver="$(dpkg-query -W -f='${Version}' ca-certificates 2>/dev/null || echo unknown)"
134+
after_fps="$(fingerprints_in_bundle)"
135+
after_count="$(grep -c . <<< "${after_fps}" || true)"
136+
if (( after_count == 0 )); then
137+
echo "::error::No certificates parsed after upgrade; cannot verify fix"
138+
exit 1
139+
fi
140+
if grep -qiF "${EMSIGN_G3_FP}" <<< "${after_fps}"; then
141+
echo "::error::emSign ECC Root CA - G3 still present after upgrade (ca-certificates ${after_ver} on ${VERSION_ID})"
142+
exit 1
143+
fi
144+
echo "ca-certificates upgraded to ${after_ver}; emSign ECC Root CA - G3 absent."

0 commit comments

Comments
 (0)