Skip to content

fix(ci): pin Windows release runner to windows-2022 - #48

Closed
tnunamak wants to merge 8 commits into
mainfrom
waspflow/dc-windows-build-fix-0902
Closed

fix(ci): pin Windows release runner to windows-2022#48
tnunamak wants to merge 8 commits into
mainfrom
waspflow/dc-windows-build-fix-0902

Conversation

@tnunamak

@tnunamak tnunamak commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Release builds for Windows stopped producing an installer. GitHub's windows-latest runner image — the label that always points at whatever Windows image GitHub currently considers current — moved to a Visual Studio 2026 install under a bare \18\ path. node-gyp, the tool npm uses to compile native C++ addons, could not parse that path and failed to build better-sqlite3, the SQLite database library this app links natively, with unknown version "undefined" found at ... \18\Enterprise. No Windows artifact was produced at all.

node-gyp 12.1.0 and later understand the new path, so the obvious fix is to make npm use a newer one. That does not work, and this was confirmed on a live run rather than reasoned about: npm_config_node_gyp, the environment variable that normally points npm at a specific node-gyp, is ignored for the automatic native rebuilds npm ci performs, because npm's run-script implementation hardcodes its own bundled copy (npm/cli#2839).

The fix is instead to pin the Windows release job to windows-2022, an image whose Visual Studio install predates the naming change. One line in .github/workflows/release.yml, one matrix entry; the macOS and Linux entries are untouched. A dated TODO records the condition for removing the pin: once actions/setup-node's Node 22 LTS bundles npm 11.6.3 or later, which brings node-gyp 12.1.0, the stock image will work again.

Pinning the runner is a workaround for a toolchain gap, so it is worth being explicit that it cannot hide a genuine build failure. The Windows job still runs scripts/verify-bundled-personal-server.mjs, which checks the produced bundle, and the artifact upload still sets if-no-files-found: error. A broken or missing installer fails the job exactly as before; the pin changes which compiler toolchain is selected, not whether the output is checked.

The failures this uncovered

Getting past node-gyp let the Windows build run further than it ever had, which exposed three latent Windows-only bugs. Each had to be fixed for the build to complete, so they ship together rather than as follow-ups.

@pdpp/polyfill-connectors — a package this repository vendors as a tarball rather than pulling from a registry, which supplies the connectors that fetch a user's data from third-party services — ran spawnSync("patchright", ...) in its postinstall hook without shell: true. Patchright is a drop-in Playwright replacement that drives a real browser while avoiding the fingerprints sites use to detect automation, and the hook installs its bundled Chromium. On Windows, a PATH lookup without a shell does not find the .cmd shim npm writes for package binaries, so this failed with ENOENT. Fixed upstream in PDP-Connect/data-connectors#60, then re-vendored here following reference-implementation/vendor/README.md, with SHA256SUMS and the package-lock.json integrity hash updated to match.

Two more vendored packages under reference-implementation/vendor/ hit the same missing-shim problem in their own build scripts, which call execFile("npx", ...) directly: read-core and mcp-server. Both now pass shell: platform === "win32". A third package in that directory, cli, needs no change because its build is written as an npm script string, and npm always runs those through a shell anyway.

In the Rust code, src-tauri/src/commands/ref_server.rs imported kill_process_group without a #[cfg(unix)] gate. That function is Unix-only at its definition and every call site was already gated, so only the import was wrong — an E0432 unresolved-import error on Windows and nowhere else.

Round 2: review findings addressed

An independent reviewer found the substance sound and raised four items, numbered F1 to F4 in their report. All four are addressed below, most significant first.

F1 — the title described an approach this PR abandoned. The title still read "pin modern node-gyp on Windows release builds" while the body explained that route was confirmed not to work. Because a squash merge takes its subject from the PR title, that misleading line would land on main as the very commit someone greps when deciding whether to unpin. Retitled to "fix(ci): pin Windows release runner to windows-2022".

F3 — a safety test had gone vacuous in CI. cdp-method-allowlist.test.ts guards the code that streams a live browser session to the user interface. It asserts that this code sends no Chrome DevTools Protocol method — the wire protocol used to control Chromium — outside an explicit allowlist, which protects an anti-detection property: the browser is driven through Patchright specifically so that sites cannot tell it is automated, and a stray Chrome DevTools Protocol call from a forbidden domain is exactly the kind of signal that gives that away. That check is vacuously true if no methods are found at all, so three assertions that specific methods are present exist to prove the extraction still finds something. A commit earlier in this same PR put all three behind a check for @opendatalabs/remote-surface, an optional dependency requiring Node 24 that is never installed on the Node 22.23.1 CI job — so in CI they never ran and the test passed regardless of what the streaming code contained. Two of the three were wrong to gate: Page.startScreencast and Input.dispatchMouseEvent are sent by this repository's own streaming code, in cdp-companion.ts, so they hold with or without the optional package. Only Page.enable lives solely in the optional package. The first two are now asserted unconditionally.

F4 — one test file was running with no type checking at all. run-interaction-stream-remote-surface-session.test.ts needs TypeScript's DOM library for its jsdom-based fake browser, but TypeScript applies that setting per compilation program rather than per file, so including it would have replaced every other file's ambient Headers and ReadableStream types with the browser versions instead of the Node ones. It was therefore excluded from the type-check program entirely — while still executing under the test runner, so a type error in it could fail nothing. A second program, tsconfig.dom.json, now covers exactly that one file with DOM enabled, and the typecheck script runs both. Type-checking the file for the first time surfaced a stale @ts-expect-error on its jsdom import, which is now an error in its own right because the types it claims are missing do resolve; removed.

F2 — scope. The reviewer noted this PR carries several concerns against the repository's "one change per PR" rule and explicitly declined to block on it, since the commits are individually clean and each one unblocked the next. Left as-is deliberately: splitting a proven, green Windows fix at this point costs more than it returns.

Verification

The strongest evidence is a direct contrast in CI on this repository: the windows-2022 build job is green on this head, while the same job on windows-latest is red on a sibling PR against the same base. The macOS and Linux legs stayed green.

Both test repairs were checked by mutation rather than just re-run, since a passing test is not yet evidence it would catch anything:

  • Renaming Page.startScreencast and Input.dispatchMouseEvent to non-allowlisted names in the streaming sources, with the optional package absent as in CI, passed before the F3 fix and now fails on the Page.startScreencast assertion.
  • A deliberate const x: number = "..." appended to the formerly-excluded test file is invisible to the original type-check program and fails the new one with TS2322. That file's runtime behaviour is unchanged — it still runs and passes.

Both mutations were reverted and the tree verified clean.

Both repairs are also confirmed live in CI on this head, which matters because each one was specifically about CI behaviour. The type-check job's log shows it now runs both programs (tsc --noEmit -p tsconfig.json && tsc --noEmit -p tsconfig.dom.json) and passes, so the formerly-excluded file is genuinely covered rather than merely coverable. The allowlist test reports test:pass on the Node 22 job — the job where the optional package is absent and where its presence assertions previously did not execute at all.

reference-implementation gate, test reference implementation, and typecheck reference implementation are three CI jobs that lint, test, and type-check the reference implementation. The first two fail on main itself, before this branch's changes, because of an unrelated in-progress migration that left the reference implementation with type errors and failing tests — tracked as issue #55, "unblock the Move-B CI seam". They are not affected by this PR. typecheck reference implementation is the exception worth noting: it fails on main and passes here, because the final commit also cleared 34 pre-existing type errors. Those were resolved by fixing type annotations and by excluding the one DOM-dependent test file described in F4 below, not by loosening any compiler option; the tsconfig's strictness settings are untouched.

Fixes #47, which reported that the Windows native build of better-sqlite3 regressed between 2026-08-21 and 2026-09-02 because node-gyp could not parse the windows-latest runner's Visual Studio 18 version string.

Assisted-by: AI

windows-latest now ships a Visual Studio install under a bare
"\18\" path (VS 2026's naming scheme). The node-gyp bundled with
Node 22.23.1's npm (10.9.8) predates VS18 detection support and
fails better-sqlite3's native build with
`unknown version "undefined" found at ... \18\Enterprise`.

Install node-gyp>=12.1.0 (nodejs/node-gyp#3240 added VS2026/18.x
detection) globally on the Windows leg only, and point npm at it
via npm_config_node_gyp, so node-gyp is resolved before any `npm
ci` triggers a native build.

Fixes #47

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
windows-latest's Visual Studio install moved to a bare "\18\" path
(VS 2026 naming). node-gyp bundled with Node 22.23.1's npm (10.9.8)
can't parse that version and fails better-sqlite3's native build
with `unknown version "undefined" found at ... \18\Enterprise`.

Tried pinning a modern node-gyp (>=12.1.0, which added VS2026/18.x
detection per nodejs/node-gyp#3240) via npm_config_node_gyp first,
but npm's run-script hardcodes its own bundled node-gyp for
automatic native rebuilds during `npm ci` and ignores that config
(npm/cli#2839) — confirmed live: the env var was set but the
rebuild still used node-gyp 11.5.0 and failed identically.

Pin to windows-2022 instead, whose Visual Studio install predates
this naming change, until actions/setup-node's Node 22 LTS bundles
npm>=11.6.3.

Fixes #47

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…ght fix

Once the windows-2022 pin unblocked node-gyp/VS18 (previous commit),
the Windows build progressed far enough to hit a second, previously-
latent bug: the vendored package's install-patchright-browser.mjs
postinstall hook failed with `spawnSync patchright ENOENT` on
Windows, because spawnSync resolves a bare command name via PATH
lookup only and misses the .cmd shim npm writes for bin entries.

Fixed upstream in PDP-Connect/data-connectors#60 (merged
cfb747b2cf17788a353d3d6e6ac1d2f5a86d9180): shell:true on win32 so
cmd.exe resolves the shim, plus a unit test of the platform-
conditional resolution logic.

Re-vendored packages/polyfill-connectors from that commit following
this directory's documented recipe (vendor/README.md): rewrite the
package's own file:./vendor/*.tgz deps on collector-runtime/
connector-protocol/reference-contract to "*" (those already exist
natively in this repo), delete the now-unused nested vendor/, npm
pack. Updated SHA256SUMS, package-lock.json's integrity hash, and
the README's provenance note to match.

Fixes #47

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
… builds

Re-vendoring @pdpp/polyfill-connectors (previous commit) unblocked the
Windows build far enough to reach these two vendored packages'
prepare/prepack -> build scripts, which both failed identically:
execFile("npx", ...) resolves the bare command name via PATH lookup
only, which misses the .cmd shim npm writes for bin entries on
Windows (no shell means no PATHEXT resolution) -- `spawn npx ENOENT`.

Same defect class and same fix as data-connectors#60 (this repo's own
vendor/read-core and vendor/mcp-server are separately-committed
copies, not that package, so the fix has to land here directly): set
shell: true on win32 only, leaving POSIX's existing direct PATH
resolution unchanged.

vendor/cli's build script runs as an npm "scripts" string (not a
direct execFile call), which npm always invokes through a shell
regardless of platform, so it isn't affected by this bug.

Fixes #47

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Once the two npx-via-shell fixes (previous commit) got the Windows
Rust build compiling, it failed with E0432: this file imports
kill_process_group unconditionally, but the function itself is
#[cfg(unix)]-gated in commands/server.rs. Every actual call site in
this file already wraps its use in #[cfg(unix)] with a
#[cfg(not(unix))] fallback (matching the established pattern already
used in commands/connector.rs) -- only the top-level `use` was
missing the same gate.

Fixes #47

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…t failures

`typecheck reference implementation` and `test reference implementation`
were both red on plain main already (confirmed via workflow_dispatch
against main, run 33702664046: typecheck failed identically before any
of this branch's changes). Fixing both at source since #47's PR must
not merge around failing checks.

typecheck (34 -> 0 errors):
- tsconfig.json: exclude test/run-interaction-stream-remote-surface-session.test.ts
  from tsc's program. @types/jsdom's index.d.ts carries
  `/// <reference lib="dom" />`, which force-included DOM lib
  project-wide despite this tsconfig's deliberate "lib": ["ES2023"] --
  replacing every ambient Headers/RequestInit/ReadableStream with the
  vanilla DOM-spec versions instead of undici-types', breaking every
  undici-fetch call site (server/cimd.ts, server/ssrf-guard.ts,
  server/source-declaration-trust/live-retrieval.ts) and every
  Buffer/BodyInit boundary. Node's test runner still executes this file
  via type-stripping regardless -- this program only ever runs
  `tsc --noEmit`, so excluding it from tsc's program doesn't change what
  runs, only stops its DOM types from leaking into every other file.
- test/pdpp-vendored-runtime-compatibility.test.ts: deleted a test
  asserting PDPP's OWN vendored tarball pins via a pdpp-repo-only script
  (scripts/check-pdpp-vendored-package-pins.ts, which only exists in
  PDP-Connect/pdpp) -- a copy-paste artifact from commit #238 that could
  never pass here. Kept the real STREAM_EVIDENCE compatibility test in
  the same file.
- runtime/browser-surface-lease-sweep-timer.ts: explicit return-type
  annotation on `options.setIntervalFn ?? setInterval` (ambiguous
  return-type inference between the option's declared signature and the
  global fallback).
- scripts/stream-health-audit/live.ts: (a) narrow local `declare const
  document` for a Playwright waitForFunction predicate that's serialized
  and executed inside the browser page it drives, not in this program;
  (b) explicit runtime assertion that auth.header.cookie is set before
  calling fetchOwnerSourcesDom -- noUncheckedIndexedAccess couldn't see
  the cross-branch guarantee the existing !auth.supported guard already
  provides.
- scripts/stream-health-audit/authority.test.ts: typed a test helper's
  revision param string | null to match its actual "simulate absent
  header" usage; cast a Json-typed fixture helper's nested field at 3
  call sites (matches this file's own established pattern elsewhere).
- scripts/test-accounting/inventory.test.ts: `delete
  missingSuite.environment_unset` instead of `= undefined` -- the test
  means "field absent," not "field present holding undefined," which is
  what exactOptionalPropertyTypes distinguishes.

test (132 of ~150 failures on main -> substantially fixed):
- test/cli.test.ts, test/cli-ref-wrapper.test.ts: both spawn
  cli/index.ts via bare `node` with no loader. Its import graph reaches
  @pdpp/reference-contract's vendored tarball, which ships raw .ts under
  node_modules -- Node's native type-stripping refuses that by policy
  (ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING). Fixed every execFile/
  spawn call site to pass `--import tsx`, matching this repo's own
  established pattern elsewhere. Confirmed via a from-scratch clone of
  main: 132 -> 2 failures of this exact error.
- server/streaming/cdp-method-allowlist.test.ts: guarded a hardcoded
  path into the optional @opendatalabs/remote-surface package (requires
  Node >=24; this CI job runs Node 22.23.1, where npm silently skips
  installing it) with existsSync, and scoped the test's "sanity check"
  assertions to only run when that package is actually present.

Remaining test failures (independently verified, not fixed -- each
confirmed identical on a from-scratch clone of main, well outside this
PR's scope):
- ~45 ENOENT failures against apps/site/*, root docker-compose.yml,
  Dockerfile, deploy/**: paths that don't exist anywhere in this repo
  yet -- tests written ahead of that content landing.
- ~58 "Promise resolution is still pending" cancelledByParent watchdog
  failures (e.g. test/controller-cancel-run.test.ts): reproduced
  identically on a completely separate, fresh clone of main run in
  isolation, so unrelated to this worktree's shared-process contention.
  A real pre-existing gap, but outside this fix's scope.
- @pdpp/polyfill-connectors ships raw .ts with no build step, crashing
  any script invoked as plain `node <file>.ts` that transitively imports
  it (compact-record-history.ts, generate-connector-registry.ts, static-
  secret generation) -- real bug, but the fix lives in
  PDP-Connect/data-connectors's packaging, not here.
- A pnpm --dir apps/console build 404 for a private package, and a
  pre-existing production-code connector-knowledge scanner backlog
  across files this PR never touched.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…face

The CDP allowlist test asserts that streaming code sends no method outside an
explicit allowlist. That check is vacuously true when no methods are extracted
at all, so three presence assertions exist to prove the extraction still finds
something -- they are the test's only defence against silently checking nothing.

Making @opendatalabs/remote-surface optional put all three behind
`if (remoteSurfaceInstalled)`. That package requires Node >=24 and is never
installed on the Node 22.23.1 CI job, so on CI the presence assertions never
ran and the suite passed no matter what the streaming code contained.

Two of the three methods are wrong to gate. Page.startScreencast and
Input.dispatchMouseEvent are sent by this repo's own cdp-companion.ts, so they
hold with or without the optional package; only Page.enable lives solely in
remote-surface's CDP backend. Assert the first two unconditionally and gate
only Page.enable.

Proof: rewriting Page.startScreencast and Input.dispatchMouseEvent to
non-allowlisted names in cdp-adapter.ts and cdp-companion.ts, with the optional
package absent, passed before this change and now fails on the
Page.startScreencast assertion.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI
Excluding run-interaction-stream-remote-surface-session.test.ts from
tsconfig.json stopped its DOM-only types leaking "DOM" lib into every other
file's ambient globals, which had to stay the undici-types-backed ones. The
cost was total: the file still runs under Node's test runner but was in no tsc
program at all, so any type error in it failed nothing.

TypeScript applies `lib` per program, not per file, so the fix is a second
program rather than a per-file setting. tsconfig.dom.json extends the base
config, adds "DOM" to `lib`, and includes only that one file. The `typecheck`
script now runs both programs, so CI covers the whole tree.

Typechecking the file for the first time surfaced a stale
`@ts-expect-error` on the jsdom import claiming jsdom has no declarations.
@types/jsdom resolves here -- it is a locked devDependency of
vendor/brand-react, so `npm ci` installs it in CI too -- which made the
directive itself an error under `noUnusedTsExpectError`. Removed.

Proof: a deliberate `const x: number = "..."` appended to that file is
invisible to `tsc -p tsconfig.json` and fails `tsc -p tsconfig.dom.json` with
TS2322. The file's runtime behaviour is unchanged; it still runs and passes.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI
@tnunamak tnunamak changed the title fix(ci): pin modern node-gyp on Windows release builds fix(ci): pin Windows release runner to windows-2022 Sep 3, 2026
@tnunamak tnunamak closed this in #61 Sep 6, 2026
@tnunamak

tnunamak commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #61 (merged): same windows-2022 pin plus the macOS bundle fix; the vendored shell:true postinstall arrived via #55's re-vendor and the npx shell + cfg(unix) fixes landed in #61.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant