fix(evaluation): do not score LCB samples 0.0 when the judge cannot run by liayan · Pull Request #432 · mlcommons/endpoints

liayan · 2026-07-29T19:31:59Z

evaluate_livecodebench_worker swallows every exception and returns False, so import/setup failures get scored as wrong answers with subset status "ok". That currently always happens in this env: lcb_runner.prompts imports anthropic (not installed) and datasets>=3 cannot load the script-based code_generation_lite dataset. Observed: 335 samples "graded" 0/335 in 83s; the same outputs score 297/349 through lcb-service. A fake zero on 349/4388 samples is enough to drop a passing DeepSeek-R1 run below the MLPerf accuracy threshold.

re-raise ImportError/FileNotFoundError/RuntimeError from the worker and the gather loop; the runner already marks the subset failed and sets complete=false
call load_lcb_benchmark() once before spawning workers so setup errors fail fast with the real traceback

Sample-level grading failures (bad generated code, timeouts) still score as incorrect. This does not fix the local path itself (deps are still missing); it just stops it from reporting a wrong score.

What does this PR do?

Type of change

Bug fix
New feature
Documentation update
Refactor/cleanup

Related issues

Testing

Tests added/updated
All tests pass locally
Manual testing completed

Checklist

Code follows project style
Pre-commit hooks pass
Documentation updated (if needed)

gemini-code-assist · 2026-07-29T19:32:01Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

github-actions · 2026-07-29T19:32:12Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

codecov-commenter · 2026-07-29T19:40:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@1df1bb2). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #432   +/-   ##
=======================================
  Coverage        ?   80.16%           
=======================================
  Files           ?      147           
  Lines           ?    19811           
  Branches        ?        0           
=======================================
  Hits            ?    15882           
  Misses          ?     3929           
  Partials        ?        0

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

arekay-nv

Thanks for this.
Can you please add some tests? The one thing to look for is whether a single bad example (one that has a bad import) will cause the entire run to fail. I believe that is the reason for swallowing the exception and just counting it as a failure.

liayan · 2026-08-04T00:35:03Z

Thanks for this. Can you please add some tests? The one thing to look for is whether a single bad example (one that has a bad import) will cause the entire run to fail. I believe that is the reason for swallowing the exception and just counting it as a failure.

Thanks — tests added in 7f391fb.

On the single-bad-example case: generated code never runs in the worker process, so it can't raise ImportError there. evaluate_livecodebench passes it to the LCB grader, which executes it in its own sandboxed subprocess. A bad import in a sample comes back as a failed test result and gets scored 0, same as before. The except Exception: return False in the worker is kept for exactly that case.

The three re-raised types can only come from the judge env itself: ImportError from the lcb_runner imports (lcb_runner.prompts imports anthropic), FileNotFoundError from the missing submodule check, RuntimeError from datasets>=3 refusing to load code_generation_lite. These hit every sample or none — that's how 335/335 got scored 0.0 in 83s. The new pre-flight load_lcb_benchmark() call catches most of this before workers even start.

Tests:

worker returns (qid, False) on a generic exception — run continues
worker re-raises each of the three infra types
one failing sample scores 0, the other samples still grade
all workers hitting an infra error raises instead of returning a fake 0/N
broken load_lcb_benchmark raises before the pool is created

1 and 3 also pass on main, so the swallow behavior is unchanged. The pool is stubbed with an in-process fake so the tests don't depend on the multiprocessing start method.

evaluate_livecodebench_worker swallows every exception and returns False, so import/setup failures get scored as wrong answers with subset status "ok". That currently always happens in this env: lcb_runner.prompts imports anthropic (not installed) and datasets>=3 cannot load the script-based code_generation_lite dataset. Observed: 335 samples "graded" 0/335 in 83s; the same outputs score 297/349 through lcb-service. A fake zero on 349/4388 samples is enough to drop a passing DeepSeek-R1 run below the MLPerf accuracy threshold. - re-raise ImportError/FileNotFoundError/RuntimeError from the worker and the gather loop; the runner already marks the subset failed and sets complete=false - call load_lcb_benchmark() once before spawning workers so setup errors fail fast with the real traceback Sample-level grading failures (bad generated code, timeouts) still score as incorrect. This does not fix the local path itself (deps are still missing); it just stops it from reporting a wrong score.

Worker tests: generic exceptions still return (qid, False), the three infra error types propagate. Pipeline tests: one failing sample scores 0 and the run continues, an infra error in a worker aborts the run, a failing load_lcb_benchmark raises before the pool is created. The pool is replaced with an in-process fake so the tests don't depend on the multiprocessing start method. Also add mlperf_eval to the mypy ignore_errors overrides: these tests are the first import of eval_accuracy, which pulls it into mypy's scope past the path exclude (same treatment as openai_types_gen and metrics/reporter). On main the infra/pre-flight tests fail and the two swallow-behavior tests pass.

… override pre-commit mypy flagged the function (annotated to return a dict, never returns; predates this PR, 1577095). It surfaced because the new tests are the first import of eval_accuracy, which put the module in mypy's scope past the path exclude. Nothing consumes a return value, so make it -> None instead of keeping the ignore_errors override.

Address review nits: hoist the constant next to the module logger, and rename the test module to test_legacy_mlperf_eval_accuracy.py to match the module under test.

Per review: note that the call both fails fast on a broken judge and warms the per-process lru_cache for fork-started workers, and that the return value is intentionally discarded.

roborluo

LGTM, Thanks for the quick revision!

liayan requested a review from a team July 29, 2026 19:31

arekay-nv reviewed Jul 31, 2026

View reviewed changes

liayan force-pushed the fix/lcb-local-eval-fail-loudly branch from 5373352 to 7f391fb Compare August 4, 2026 00:05

liayan force-pushed the fix/lcb-local-eval-fail-loudly branch from 7f391fb to 1eabc9f Compare August 4, 2026 01:29

liayan added 2 commits August 4, 2026 10:26

liayan force-pushed the fix/lcb-local-eval-fail-loudly branch from 1eabc9f to 66037f0 Compare August 4, 2026 14:26

nvzhihanj requested a review from roborluo August 4, 2026 18:33