fix(cli): forward solver config to all model benchmarks and improve error resilience

[devin-ai-integration]

TL;DR

Community fix from @warun7 (authored commit preserved, brought in from warun7:benchmark-harness:fix/cli-solver-config-and-resilience because the repo's interaction limits blocked a direct PR): --solver-config is now honored for gpqa_diamond, mmlu_pro, mmmu_pro_vision, and ifstruct, the progress bar is always stopped on failure, and non-numeric values for numeric flags fail fast.

What changed?

• buildBenchmarkConfig no longer hand-builds configs for gpqa_diamond, mmlu_pro, mmmu_pro_vision, ifstruct, tau_bench_verified_airline, tau3_bench_banking, terminal_bench. All fall through to buildSchemaValidatedConfig, which merges panelConfig keys (providerOnly, maxTokens, timeoutMs, sort, ...) and validates against BenchmarkRunConfigSchema. buildSchemaValidatedConfig gains an optional imageDetail passthrough for mmmu_pro_vision.
• runBenchmarkById call in main is wrapped in try ... finally so bar.stop() runs (only if the bar was started) when the run throws, restoring the terminal cursor.
• num() in parseArgs throws TypeError("<flag> must be a valid number, got: <raw>") when Number(raw) is not finite, instead of returning NaN into --concurrency, --limit, --start, --end, --epochs.
• Tests added in src/cli/index.test.ts for all three behaviors.

Why?

Reported on Discord by warun7. On main, the four model benchmarks above parsed --solver-config but never used it, so provider pinning and token/timeout limits silently did nothing. A crash mid-run left the terminal cursor hidden. --concurrency abc produced NaN and undefined downstream behavior.

Searched existing PRs (open, closed, draft) for "solver-config", CLI, and src/cli/index.ts; none matched.

How to test

bun run bench --benchmark gpqa_diamond --model openai/gpt-5 --solver-config '{"providerOnly":["together"],"maxTokens":2048}' --limit 1

Expected: request goes to the pinned provider with max_tokens: 2048 (on main it did not).

bun run bench --benchmark gpqa_diamond --model openai/gpt-5 --concurrency abc

Expected: exits with --concurrency must be a valid number, got: abc.

bun run bench --benchmark gpqa_diamond --model openai/gpt-5 --solver-config '{"unknownOption":true}'

Expected: Unknown gpqa_diamond solver-config option(s): unknownOption (previously silently ignored).

Locally on this branch: bun run format:check, bun run check, bun run typecheck, bun test (1409 pass), bun run build all succeed.

Benchmark impact

No change to solvers, scorers, prompts, or datasets. Runs of the four affected benchmarks that passed a --solver-config will now actually apply it, so results from such invocations are not comparable to pre-fix runs where the config was dropped. Runs without --solver-config are unaffected.

Reviewer focus

• Stricter behavior: unknown --solver-config keys now throw for gpqa_diamond, mmlu_pro, mmmu_pro_vision, ifstruct instead of being ignored. This matches how the other model benchmarks already behaved.
• imageDetail is passed only for mmmu_pro_vision, keeping other configs schema-identical to before.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing (n/a)
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed (n/a)

Link to Devin session: https://openrouter.devinenterprise.com/sessions/e348d039d09045f3a3cc9fc485bf2cda
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/e348d039d09045f3a3cc9fc485bf2cda?variant=devin
Requested by: @jasoncrabtree-or

---

Note

Devin errored when opening this Pull Request as jasoncrabtree-or.
As a fallback, Devin opened this PR as itself.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Jason

help me with this report from a discord user:

Hey team! I was going through ``@openrouter/bench-harness and found a few bugs in the CLI runner. I put together a fix with full test coverage:

1. **`--solver-config` ignored**: In `src/cli/index.ts`, `gpqa_diamond`, `mmlu_pro`, `mmmu_pro_vision`, and `ifstruct` dropped `panelConfig`. Flags like `providerOnly`, `maxTokens`,

timeoutMs, and sort were parsed but discarded. I routed them through buildSchemaValidatedConfig
2. Terminal cursor hidden on crash: Wrapped runBenchmarkById in try ... finally so bar.stop() always runs if an error occurs
3. NaN numeric flag handling: Added finite number checks for --concurrency, --limit, etc., to avoid NaN values

The repo has interaction limits on, so I could not open a PR directly. The branch and tests (1,409 passing) are ready on my fork:
👉 https://github.com/OpenRouterTeam/benchmark-harness/compare/main...warun7:benchmark-harness:fix/cli-solver-config-and-resilience

I also have a fix ready for HTTP 408 timeouts being scored as `Incorrect` instead of `Skipped` if helpful!