Model availability reports `online` when the provider API key is invalid — then a whole run scores 0

[albinekb]

Version: BenchLocal 0.3.0 (Agent API on 127.0.0.1:1232)
Provider kind: openai_compatible → https://openrouter.ai/api/v1

Summary

POST /v1/tabs/:tabId/models/availability/refresh reported status: "online", reason: "available" for all 22 models configured against a provider whose API key had been revoked. The subsequent run then failed all 330 scenarios in ~2.5s total, each recorded as a scored result (score: 0) with errorType: "provider_error" and providerHttpStatus: 401.

The result is a benchmark grid that is entirely zeros and looks, at a glance, exactly like a set of very bad models.

Why it happens

The availability probe appears to be a GET {base_url}/models listing. On OpenRouter that endpoint is unauthenticated — it returns HTTP 200 and the full model catalogue regardless of whether the bearer token is valid, expired, or absent. So the probe confirms only "this host is reachable and serves these ids", never "these credentials work".

This isn't OpenRouter-specific: any OpenAI-compatible gateway with a public model list has the same hole.

Reproduction

1. Add a provider: POST /v1/providers with
   {"kind":"openai_compatible","name":"openrouter","base_url":"https://openrouter.ai/api/v1","api_key":"<any invalid key>"}
2. Register a model against it, e.g. minimax/minimax-m2.
3. Create a tab, select the model, then POST /v1/tabs/:tabId/models/availability/refresh.

Expected: the model is offline / auth_failed, or at minimum not reported as verified.

Actual:

{"modelId":"…:minimax/minimax-m2","status":"online","reason":"available","checkedAt":"…"}

4. Start a run. Every scenario returns in ~450ms:

{"status":"fail","score":0,"summary":"User not found.",
 "verifier":{"status":"fail","summary":"Provider returned an HTTP error status.",
             "details":{"providerHttpStatus":401,"providerResponseBlank":false}},
 "errorType":"provider_error","retryable":false}

("User not found." is OpenRouter's message for a revoked key or deleted account.)

Suggested fixes

Either would have saved the confusion; the second matters more.

1. Make the probe prove authentication. A model listing can't. Options: send the credential and treat 401/403 as auth_failed; probe a chat/completions with max_tokens: 1; or keep the cheap probe but report a distinct status (reachable_unverified) so the UI doesn't claim green.

2. Fail a run fast on an auth error instead of scoring it. A 401 with retryable: false is a configuration fault, not a model result. Recording it as score: 0 per scenario silently manufactures a full grid of plausible-looking failures. Aborting the run with "provider authentication failed" would make the cause obvious immediately.

This is the same family as #7 ("fetch failed is actually a timeout abort misclassified") and #13 (timeouts surfacing as provider errors): a transport/config failure becomes indistinguishable from a wrong answer once it's written into a score cell. A general rule like "errorType: provider_error never produces a scored result" would cover all three.

Two smaller things found alongside (happy to file separately)

a) A 429 mid-tool-loop scores correct work as zero. Running 22 models concurrently on ToolCall-15, one model got throttled upstream partway through a multi-turn scenario. It had already completed the search → read → email chain correctly; the next call returned 429, and the scenario was recorded score: 0, summary: "Provider returned error". Retries (3) all hit the same throttle. Same root cause as above: provider errors becoming scores.

b) Cancelling a run writes cancellation into result cells. After POST /v1/tabs/:tabId/runs/stop, in-flight scenarios were saved as status: "fail", score: 0 with rawLog containing error=Run cancelled by user. In the UI these are visually identical to genuine failures — one model showed a complete row of 15 red cells at a uniform 11s. A distinct cancelled status (excluded from scoring) would prevent misreading a stopped run as a bad model.

c) Bonus feature note: result objects carry no token counts (score, points, status, summary, rawLog, timings, verifier, errorType, retryable, note), so cost-per-point can't be computed from a run even though upstream APIs return usage on every response. Recording usage when the provider supplies it would make price/performance comparison possible — which is the main thing I was trying to measure.