fix(datasets): send HF token, retry only 429/5xx, honor Retry-After for cached file fetches

[devin-ai-integration]

TL;DR

fetchCachedTextFile (the path tau-bench-airline db.json and the tau3 banking files take on a cache miss) now sends HF_TOKEN to huggingface.co, retries only on 429/5xx/transport errors, and honors Retry-After. Follow-up to #75.

What changed?

• src/datasets/cached-file.ts
  • CachedTextFileRequest.hfToken?: string. When omitted, the helper reads HF_TOKEN through resolveHfToken() (same source as the generic HF rows loader). The Authorization: Bearer header is sent only when the URL host is huggingface.co or a subdomain, so the token never reaches the GitHub raw URLs used by tau3 banking.
  • CachedFileError.retryAfterMs?: number, parsed from the retry-after response header (delay-seconds or HTTP-date).
  • isRetryableCachedFileFailure: retry CachedFileError only for status 429 or >= 500; always retry HttpClientError (transport / body read). A 404 or 403 now fails fast instead of burning the retry budget.
  • Cache key is unchanged (files/<url>.json), so entries already in the GCS bucket stay valid.
• src/datasets/huggingface.ts
  • hfFetchRetrySchedule(config, isRetryable, retryAfterMs?): optional third parameter that overrides the computed backoff with the server-provided delay. Applied before withRetryAttemptLogging, so delay_ms in the retry log reflects the real wait. Existing callers are unaffected (defaults to undefined → computed delay).
• Tests
  • src/datasets/cached-file.test.ts: non-retryable 404 makes exactly one request; 503 retries; retry-after: 1 delays the retry by >= 900 ms; token header present only for huggingface.co and absent for an empty token; unit tests for isHuggingFaceUrl and parseRetryAfterMs. Existing cache-hit and 429-then-success tests still cover those paths.
  • src/benchmarks/tau-bench-airline/airline.test.ts: ensureAirlineData makes no network request once the in-process copy is populated.

Why?

Datadog APM, 7 days ending 2026-09-09, production bench-worker: 51,556 of 87,052 errored RunActivity:runBenchmark attempts (59%) failed with Failed to load airline data from HF: ... Failed to fetch db.json from HF (429). Each Temporal activity retries up to 5 times, so the same chunk keeps hitting the same 429.

Burst amplification: the pre-#75 loader memoized db.json in a module-level variable, which is per process. The worker pool runs one activity per Cloud Run instance and scales to roughly 1000 instances, so every scale-up wave is a burst of identical anonymous resolve/.../db.json downloads from fresh processes, with no shared cache, no backoff, and no token. Hugging Face rate-limits the anonymous burst, and the Temporal retry multiplies each 429 by five.

#75 (merged, not yet pulled into openrouter-web) moved db.json onto the shared cache store and added backoff, which removes the steady-state burst. This PR closes the remaining gaps in that path so a cold-cache burst after a revision bump behaves like the generic HF loader:

1. Authenticated requests get the higher HF per-token rate limit instead of the anonymous per-IP limit shared by all Cloud Run egress.
2. Retry-After from HF is honored instead of guessed.
3. Non-retryable statuses fail fast and leave the retry budget for 429/5xx.

Searched existing PRs (last 90 days, harness and openrouter-web): #75 covers the cache-store routing; nothing covers the token, retry classification, or Retry-After. openrouter-web #40710 was a direct subtree edit and is closed.

How to test

bun run format:check
bun run check
bun run typecheck
bun test
bun run build

All five pass locally (1434 tests, 0 failures). Targeted run:

bun test src/datasets/cached-file.test.ts src/benchmarks/tau-bench-airline/airline.test.ts

Expected: retry-after test logs delay_ms: 1000 in the Retrying after transient error line; the 404 test logs no retry.

Benchmark impact

None. Dataset content, revision pin, solver, and scorer are unchanged. Only the transport for fetching the environment file changes.

Reviewer focus

• isRetryableCachedFileFailure treats every HttpClientError as retryable. Transport errors and body-read failures are the only members of that union on this path.
• authorizationHeaders host check: huggingface.co and *.huggingface.co only.
• hfFetchRetrySchedule third parameter is additive; existing call sites pass two or fewer arguments.

Follow-up

• openrouter-web: run scripts/subtree-pull-bench-harness.sh to pull #75 and this PR into packages/bench-harness. Production still runs the pre-fix: cache tau environment files in the dataset cache store #75 loader until that lands; the Datadog figures above are all from that code.
• Cold-cache concurrency across chunks is still uncoordinated (one fetch per chunk after a revision bump), as noted in fix: cache tau environment files in the dataset cache store #75. Authenticated requests plus Retry-After make that burst survivable; a cross-process lock is out of scope here.

How to test in prod

• What to verify: after the openrouter-web subtree pull deploys the bench worker, tau-bench-airline chunks stop failing with Failed to fetch db.json from HF (429) / HTTP 429 for https://huggingface.co/datasets/abhinavpola/tau2-bench-verified-airline/....
• Datadog APM: filter service:bench-worker resource_name:"RunActivity:runBenchmark" status:error over the 24 h after deploy and search the error message for 429. Success criterion: 429-tagged errors drop to near zero as a share of RunActivity:runBenchmark errors (baseline 59%). Query: https://us5.datadoghq.com/apm/traces?query=service%3Abench-worker%20resource_name%3A%22RunActivity%3ArunBenchmark%22%20status%3Aerror%20%40error.message%3A%2A429%2A
• Datadog logs: search service:bench-worker "Retrying after transient error" "CachedFileError". Expected: sparse entries with delay_ms equal to the HF Retry-After value when present; no retries with error_message: "HTTP 404 ...".
• GCS: gsutil ls gs://<bench dataset cache bucket>/dataset-cache/files/ | grep tau2-bench-verified-airline returns one object for the pinned revision 790bdd0336f4e3386824ced48ee2a98a11058345 (bucket name from google_storage_bucket.bench_dataset_cache in services/gcp-bench-worker/cloudrun/).
• Rollback signal: 429 share of RunActivity:runBenchmark errors stays at or above baseline, or new HTTP 401/HTTP 403 errors for huggingface.co appear (indicates a bad GCP_BENCHMARKING_HF_TOKEN).

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing (not applicable: no dataset change)
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed (not applicable)

[Used playbook: Google Developer Docs Style — Write or Review Docs]

Link to Devin session: https://openrouter.devinenterprise.com/sessions/0cc065effe934a91a31f6f7c7fa1fce2
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/0cc065effe934a91a31f6f7c7fa1fce2?variant=devin
Requested by: @jamespsterling

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Devin Bot

Fix Hugging Face 429 rate-limit failures in the tau-bench-airline benchmark (bench harness).

#``# Evidence (Datadog APM, 7 days ending 2026-09-09, production bench-worker)

• 51,556 of 87,052 errored RunActivity:runBenchmark attempts (59%) fail with Failed to load airline data from HF: ... Failed to fetch db.json from HF (429).
• Each Temporal activity retries up to 5 times, so the same chunk keeps hitting the same 429.
• Root cause: packages/bench-harness/src/benchmarks/tau-bench-airline/environment.ts (fetchHfFile / ensureAirlineData) fetches https://huggingface.co/datasets/abhinavpola/tau2-bench-verified-airline/resolve/main/db.json directly with a plain HttpClient.get. Its cache is a module-level variable (per process), there is no retry/backoff, no HF token, and no shared cache. The worker pool runs 1 activity per Cloud Run instance and scales to ~1000 instances, so every scale-up wave is a burst of identical downloads from a fresh process.
• The generic dataset loader packages/bench-harness/src/datasets/huggingface.ts already has retry, hfToken, and a cacheStore abstraction (packages/bench-harness/src/datasets/cache-store.ts or similar — verify). The airline db.json path bypasses all of it. The worker also provisions a GCS dataset-cache bucket (services/gcp-bench-worker/cloudrun/dataset-cache-bucket.tf in openrouter-web) — check how the generic loader's cache store is wired to it in the worker and whether the same wiring can serve db.json.

#``# Where the code lives
packages/bench-harness in OpenRouterTeam/openrouter-web is a git subtree pulled from OpenRouterTeam/benchmark-harness. Direct edits in openrouter-web fail the CI subtree-integrity check (scripts/check-subtree-integrity.ts). Land the fix in OpenRouterTeam/benchmark-harness (find the same file paths there, under src/benchmarks/tau-bench-airline/). If you cannot access that repo, stop and report that immediately instead of editing the subtree in openrouter-web. Do not do ... (2725 chars truncated...)

[abhinav-pola]

SGTM