[Bug]: VLM (qwen3-235B) scoring fails: `inference_endpoint/evaluation/scoring.py` msgspec.DecodeError: JSON is malformed: invalid character (byte 145)

[ai-nikolai]

Bug Description

VLM Benchmark: Scoring breaks

When running the VLM benchmark it fails during accuracy scoring.

Steps to Reproduce

1. Running vllm server:

vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
        --tensor-parallel-size 8 \
        --attention-backend TRITON_ATTN \
        --no-enable-prefix-caching \
        --max-model-len=32768 \
        --async-scheduling \
        --limit-mm-per-prompt.video 0 \
        --host 0.0.0.0 \
        --port 30000 &

2. Running benchmark: (using this official qwen3-vl-235B-offline config: https://github.com/mlcommons/endpoints/blob/381d13bbd27d6d52306813a51dc4e44295222d7e/examples/08_Qwen3-VL-235B-A22B_Example/offline_qwen3_vl_235b_a22b_shopify.yaml)

inference-endpoint benchmark from-config \
-c offline_qwen3_vl_235b_a22b_shopify.yaml

Environment

Docker: vllm/vllm-openai:v0.24.0-cu129-ubuntu2404

Installation of endpoints (inside docker):

git clone https://github.com/mlcommons/endpoints.git
cd endpoints
pip install .

Relevant Logs

2026-07-23 13:48:56,586 - inference_endpoint.commands.benchmark.execute - INFO - Partial results saved to engines/mlperf/results/qwen3_vl_235b_a22b_shopify_8k_benchmark_offline
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/main.py", line 132, in run
    app.meta()
  File "/usr/local/lib/python3.12/dist-packages/cyclopts/core.py", line 1889, in __call__
    result = _run_maybe_async_command(command, bound, resolved_backend)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/cyclopts/_run.py", line 50, in _run_maybe_async_command
    return command(*bound.args, **bound.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/main.py", line 73, in launcher
    app(tokens)
  File "/usr/local/lib/python3.12/dist-packages/cyclopts/core.py", line 1889, in __call__
    result = _run_maybe_async_command(command, bound, resolved_backend)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/cyclopts/_run.py", line 50, in _run_maybe_async_command
    return command(*bound.args, **bound.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/commands/benchmark/cli.py", line 183, in from_config
    _run(resolved, [], test_mode, accuracy_only=accuracy_only)
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/commands/benchmark/cli.py", line 74, in _run
    run_benchmark(config, mode)
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/commands/benchmark/execute.py", line 1681, in run_benchmark
    finalize_benchmark(ctx, bench)
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/commands/benchmark/execute.py", line 1551, in finalize_benchmark
    accuracy_scores = _score_accuracy(ctx, result)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/commands/benchmark/execute.py", line 1429, in _score_accuracy
    score, n_repeats = scorer_instance.score()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/evaluation/scoring.py", line 1359, in score
    df = self.get_scoring_outputs()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/evaluation/scoring.py", line 196, in get_scoring_outputs
    return self.get_raw_outputs()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/evaluation/scoring.py", line 179, in get_raw_outputs
    rows = [
           ^
  File "/usr/local/lib/python3.12/dist-packages/inference_endpoint/evaluation/scoring.py", line 166, in _iter_complete
    record = decoder.decode(stripped)
             ^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: invalid character (byte 145)

Before submitting

• I searched existing issues and found no duplicates

[wangshangsam]

@ai-nikolai if you could dump and provide the stripped string that triggered this msgspec.DecodeError, that would be very helpful for us to debug this issue.

Assigning to @Victor49152

[ai-nikolai]

@wangshangsam - thank you for getting back on this.

I will try to provide more info on this.... However, it might be a bit hard for me to pinpoint the exact string as it happens already inside endpoints... Do you know how I could extract the exact datapoint information for that particular failing sample? (maybe accessing the events json somehow)?