Skip to content

fix(install): authenticate api.github.com call to avoid 60/hr rate limit - #1157

Merged
backnotprop merged 3 commits into
backnotprop:mainfrom
tbontb-iaq:fix/install-github-rate-limit
Jul 31, 2026
Merged

fix(install): authenticate api.github.com call to avoid 60/hr rate limit#1157
backnotprop merged 3 commits into
backnotprop:mainfrom
tbontb-iaq:fix/install-github-rate-limit

Conversation

@tbontb-iaq

@tbontb-iaq tbontb-iaq commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1156.

All three installers (install.sh, install.ps1, install.cmd) called api.github.com/repos/.../releases/latest unauthenticated to resolve the latest release tag. GitHub caps unauthenticated REST API requests at 60/hour per source IP (shared across NAT/CGNAT/corporate proxy users, not per user), causing opaque Failed to fetch latest version errors on shared egress IPs and during repeated/debug runs within an hour.

Fix

Each installer now detects a token and attaches it as an Authorization: Bearer header to only the version-resolution API call:

Precedence Source
1 GITHUB_TOKEN env var
2 GH_TOKEN env var
3 gh auth token (if gh CLI is installed and authenticated)

When no token is available, the header is omitted and behavior is unchanged (anonymous, 60/hr) - fully backward compatible. With a token the limit rises to 5000/hour.

Scope of auth: only the api.github.com version-resolution call is authenticated. Release-asset downloads (github.com/.../releases/download/...) and git clone are intentionally left unauthenticated - they are not subject to the 60/hr REST limit, and authing large binary downloads would expose the token in the process argument list for the full download duration.

Hardening (from pre-PR self-review)

  • Token variables (_gh_token/GH_AUTH_HEADER in sh, GH_TOKEN_VAL/GH_AUTH_HEADER in cmd, $ghToken/$ghHeaders in ps1) are cleared from memory immediately after the API call - they are not needed for the download/clone phases. Defense in depth against environment leakage.
  • install.ps1 wraps the Invoke-RestMethod call in try/catch, surfacing a friendlier error (with a pointer to Installer fails behind rate-limited IPs: api.github.com called without auth (60 req/hr) #1156) instead of a raw terminating exception under $ErrorActionPreference = "Stop" (the existing anonymous path had this latent issue too; the PR widens the failure surface since token-bearing calls can 401/403).

Known limitation

The token is passed to curl via -H in install.sh and install.cmd (PowerShell uses a request-object hashtable, which is not affected). This means the token is briefly visible in the process argument list (/proc/<pid>/cmdline on Linux, ps on macOS) for the duration of the single API call (<1s). This matches the approach used by rustup, homebrew, and other installers. A more robust fix would use curl --config with a 0600 temp file, but that adds complexity (temp-file lifecycle, mktemp failure under set -e, and is impractical in batch for install.cmd). If maintainers prefer the --config approach for the POSIX installer, I'm happy to follow up.

Docs

Tests

New tests in install.test.ts (install shared behavior describe block):

  1. Parity test - asserts all three installers read GITHUB_TOKEN/GH_TOKEN, use a Bearer scheme, fall back to gh auth token, and attach the header to the api.github.com call (with per-installer mechanism assertions: array splat for sh, hashtable for ps1, delayed-expansion arg for cmd).
  2. Negative-assertion tripwire - asserts the auth header is not splatted into the binary download or checksum download curls (prevents a future edit from accidentally exposing the token during large downloads), and asserts the token variables are cleared after the API call.

All 88 tests pass (bun test scripts/install.test.ts).

Verification

$ bash -n scripts/install.sh  # syntax OK
$ bun test scripts/install.test.ts  # 88 pass, 0 fail
# Smoke test: auth header injected (2 args), tag resolved (v0.25.1), vars cleared after unset (count=0)

Out of scope

  • apps/marketing/public/install.sh and install.ps1 are git-tracked symlinks to scripts/install.* - no separate edit needed.
  • install.cmd has no public mirror (served differently).

AI assistance disclosure

This pull request was developed with the assistance of an AI coding agent (opencode). The issue investigation, code changes across all three installers, the test additions, and the self-review were produced collaboratively with the AI. I reviewed and verified every change before submitting, including running the full test suite (89 pass) and live smoke tests of the token-detection and anonymous-fallback logic (valid token succeeds first try; bad token 401s and retries anonymously).

All three installers called api.github.com/repos/.../releases/latest
unauthenticated to resolve the latest release tag. GitHub caps
unauthenticated REST API requests at 60/hour per source IP (shared
across NAT/CGNAT/corporate proxy users), causing opaque 'Failed to
fetch latest version' errors.

Each installer now detects a token (precedence: GITHUB_TOKEN env >
GH_TOKEN env > 'gh auth token') and attaches it as an Authorization:
Bearer header to ONLY the version-resolution API call. Falls back to
anonymous (unchanged behavior) when no token is available. Release
downloads and git clone are left unauthenticated - they are not subject
to the 60/hr REST limit.

Token variables are cleared from memory immediately after the API call
(defense in depth). install.ps1 wraps the API call in try/catch to
surface a friendlier error instead of a raw terminating exception under
$ErrorActionPreference=Stop.

Docs updated with a rate-limited-IP install section and a troubleshooting
entry. Parity + negative-assertion tests added.

Closes backnotprop#1156
@backnotprop

Copy link
Copy Markdown
Owner

Thanks for the fast follow-up. Reviewed all three scripts plus the tests. The token handling is scoped correctly: the header only ever reaches the hardcoded api.github.com URL, downloads stay anonymous, and the tripwire tests lock that in. Docs read well too.

One real problem found while testing, and it blocks merge as-is:

A stale or invalid token now breaks installs that work today. I verified against the live API: a bogus Bearer token gets a 401, curl -f fails, and the installer exits with the same opaque error this PR is trying to eliminate. Expired GITHUB_TOKENs linger in CI images, dotfiles, and direnv setups all the time. Those users currently install fine anonymously; with this change they fail.

Fix: if the authenticated call fails, retry once without the header before giving up. Roughly, in install.sh:

latest_tag=$(curl -fsSL "${GH_AUTH_HEADER[@]}" "$api_url" | grep '"tag_name"' | cut -d'"' -f4) || true
if [ -z "$latest_tag" ] && [ ${#GH_AUTH_HEADER[@]} -gt 0 ]; then
    latest_tag=$(curl -fsSL "$api_url" | grep '"tag_name"' | cut -d'"' -f4)
fi

with the same retry in install.ps1 (catch, then retry with empty headers) and install.cmd. That keeps the upgrade strictly additive: a good token raises the limit, a bad token costs one extra request, nobody is worse off.

Two smaller notes:

  • install.sh checked out on macOS bash 3.2: no set -u, arrays already used elsewhere, so the empty array expansion is safe. Verified with a valid token end to end.
  • The install.cmd quoting pattern reads correct but I could not run it on a real Windows box; the CI tests are source scans. If you have Windows access, one manual run of the latest-version path with and without GITHUB_TOKEN set would be good confirmation.

Happy to merge once the anonymous fallback is in.

Addresses review feedback on backnotprop#1157: a stale/revoked token (expired
GITHUB_TOKEN lingering in CI images, dotfiles, direnv) gets a 401 and
would break an install that works fine anonymously today.

Each installer now retries the api.github.com version-resolution call
without the auth header when the authenticated attempt fails:
- install.sh: `|| true` + `[ -z "$latest_tag" ]` gate
- install.ps1: try/catch, retries with no -Headers on catch
- install.cmd: ERRORLEVEL check, retries without !GH_AUTH_HEADER!

A bad token costs one extra request; nobody is worse off than today.
New test asserts the retry path exists in all three installers (89 pass).

Also corrects the AI disclosure: the assistant was opencode, not Claude.
@tbontb-iaq

tbontb-iaq commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review - the stale-token regression is a real gap, good catch.

Pushed the anonymous fallback to all three installers:

  • install.sh: || true swallows the authed curl failure, then [ -z "$latest_tag" ] && [ ${#GH_AUTH_HEADER[@]} -gt 0 ] gates an anonymous retry.
  • install.ps1: try/catch around Invoke-RestMethod; on catch, when a header was present (.Count -ne 0), retries with no -Headers. If there was no header to begin with, it surfaces the friendly error (rate-limit pointer to Installer fails behind rate-limited IPs: api.github.com called without auth (60 req/hr) #1156) and exits.
  • install.cmd: if !ERRORLEVEL! neq 0 if defined GH_AUTH_HEADER ( retries the curl without the header arg.

Verified: valid token succeeds first try (tag v0.25.1, no retry); bogus token 401s, falls back to anonymous. (My test IP had already burned its anonymous 60/hr quota during this session, so the anonymous retry 403'd locally - environmental, not a code issue; the retry path itself fired correctly.)

On the Windows note: I don't have Windows access either, so install.cmd remains source-scan-only verified. The quoting pattern and the new ERRORLEVEL-based retry follow the existing cmd idioms in the file. If CI runs the install.cmd path on Windows that would be the real confirmation; otherwise a maintainer with Windows could spot-check.

Test count is now 89 (added a retry-path parity test).

@backnotprop

Copy link
Copy Markdown
Owner

Verified the update end to end:

  • Ran the fallback logic against the live API with a deliberately bogus GITHUB_TOKEN: the authed call gets a 401, the anonymous retry fires, and the tag resolves correctly.
  • All 89 install-script tests pass locally on your branch, including the new retry tripwires.
  • macOS system bash 3.2 parses install.sh cleanly.
  • The install.cmd errorlevel flow checks out on read: a failed authed call retries, and the existing failure check after the block sees the retry's result.

CI approved and running. This is good to go from my side; final merge call is the maintainer's, likely in the morning (US time). The hosted copies at plannotator.ai are synced separately after merge, so the fix goes live with that step. Thanks for turning this around quickly.

@backnotprop

Copy link
Copy Markdown
Owner

A deeper second-pass review found real issues, so this is not merge-ready yet after all. Findings, worst first:

Blocking

  1. install.cmd: the two set clears now sit between curl and its ERRORLEVEL check. Every other ERRORLEVEL check in that file (all 12) is immediately adjacent to its command, and set can disturb ERRORLEVEL on some paths (notably set "VAR=" on an undefined variable). Depending on the Windows build this either lets a failed curl slip past the guard or fails a successful anonymous install. Fix is free either way: move the clears below the failure check, or capture set "_RC=!ERRORLEVEL!" right after curl and test that.

Security (both cmd-only, both one-line fixes)

  1. install.cmd: set "GH_TOKEN_VAL=%GITHUB_TOKEN%" is command injection. Percent expansion happens before metacharacter parsing, so a token value containing "&...& executes arbitrary commands. Use delayed expansion instead: set "GH_TOKEN_VAL=!GITHUB_TOKEN!". (This repo documents exactly this hazard elsewhere in the same file.) Relatedly, a " in the token can also split the curl argv and inject flags; cheapest guard is to reject tokens containing a quote before building the header.

  2. install.cmd: gh resolves from the current directory first. cmd searches CWD before PATH for the for /f ... ('gh auth token') invocation, and our documented Windows flow runs install.cmd from the download directory. A planted gh.exe there would now execute. Guard with where gh >nul 2>&1 or resolve an absolute path first.

Should fix

  1. Retry on any failure doubles the damage in the exact scenario this PR targets. Requests with invalid credentials count against the anonymous per-IP 60/hr pool, so a stale token behind shared NAT now burns two requests per attempt instead of one. My earlier sketch was too loose here; retry only on 401 (curl: capture %{http_code}; PowerShell: check $_.Exception.Response.StatusCode).

  2. gh auth token is not host-scoped. A user logged into GitHub Enterprise gets their enterprise token silently sent to api.github.com. Use gh auth token --hostname github.com in all three scripts.

  3. install.ps1 catch discards the original exception, so DNS/proxy/TLS failures all report as possible rate limiting. Append $_.Exception.Message. Also: when a token is present and both attempts fail, the retry throws raw with no friendly message at all.

  4. The "clear token from memory" comments overclaim. The env vars themselves stay in the environment and are inherited by git clone and gh attestation verify later in the script, so the local clears buy nothing. Either drop the claim or actually unset GITHUB_TOKEN/GH_TOKEN after the call.

  5. Four test assertions are vacuous (they match pre-existing or wrong occurrences: the || true, the ps1 substring that matches the authed call, and both cmd set clears which match the init/else lines instead). The PowerShell retry currently has no real coverage. Also consider asserting GITHUB_TOKEN-before-GH_TOKEN precedence and that the header identifier never appears after the version-resolution block, which is more robust than byte-exact negative matches.

  6. Docs: "5000/hour" is wrong for the CI case the docs lead with. The Actions GITHUB_TOKEN is capped at 1000/hr per repo; 5000 is the PAT number. Also worth stating that a zero-scope token is sufficient (public repo), rather than implying a broad classic PAT, and troubleshooting.md says --version where PowerShell takes -Version.

The scope/provenance side came back fully clean: diff touches exactly the stated files, every URL verified character by character, no non-ASCII tricks, no new hosts or subprocesses beyond gh auth token, install.sh and install.ps1 injection-safe as written.

Sorry for the two-round review; the cmd items and the 401-only retry are the substance. Happy to re-verify quickly once pushed.

Errorlevel check adjacency in install.cmd, delayed-expansion token
reads, token charset guard, PATH-resolved gh with --hostname
github.com, 401-only anonymous retry in install.sh and install.ps1,
preserved original errors in install.ps1, honest cleanup comments,
non-vacuous test assertions, and doc corrections (1000/hr Actions
token, zero-scope guidance, -Version for PowerShell).
@backnotprop

Copy link
Copy Markdown
Owner

Heads up @tbontb-iaq: rather than another review round trip, we pushed the fixes directly to your branch as 02a0ddd (maintainer edits are enabled on the PR). It addresses everything from the second review: errorlevel check adjacency in install.cmd, delayed-expansion token reads plus a charset allowlist, PATH-resolved gh invoked with --hostname github.com, 401-only anonymous retry in install.sh and install.ps1 (cmd keeps the simpler retry, with a comment), preserved original error messages in install.ps1, honest cleanup comments, non-vacuous test assertions, and the doc corrections (1,000/hr for the Actions token, zero-scope guidance, -Version for PowerShell).

Verified: 90/90 install tests pass, macOS bash 3.2 parses and runs the new block correctly (bogus token falls back on 401 and resolves; no token resolves in one call; network failure does not retry). install.cmd is verified by careful read only, since we cannot execute cmd here, so if you have a Windows box handy, one run each with and without GITHUB_TOKEN set would be a welcome final check.

Please pull before making further changes to avoid a divergent branch. Thanks again for the contribution and the quick turnarounds.

@backnotprop
backnotprop merged commit 07c89ff into backnotprop:main Jul 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installer fails behind rate-limited IPs: api.github.com called without auth (60 req/hr)

2 participants