Add Pyxis runtime for SWE-bench accuracy

[hvagadia]

Summary

• add an optional Pyxis runtime to the SWE-bench accuracy service
• run mini-swe-agent tool commands and SWE-bench evaluation through srun and Enroot on a retained one-node Slurm allocation
• resolve task images from a registry prefix using the documented sweb.eval.arm64.<instance_id>:v4.1.0-arm64 convention
• preserve Docker as the default runtime and leave the benchmark client configuration unchanged

Why

ARM64 Slurm nodes provide Pyxis and Enroot but may not provide Docker. This allows the existing external SWE-bench accuracy service to run agent and evaluator containers natively on those nodes without changing its HTTP API or the endpoint client YAML.

Validation

• uv run pytest -q tests/unit/evaluation/swebench_service — 72 passed
• targeted pre-commit checks for all five changed files — passed

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.55128% with 17 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@1df1bb2). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...ench_service/swebench_service/pyxis_environment.py	95.41%	6 Missing ⚠️
.../swebench_service/swebench_service/pyxis_worker.py	94.82%	6 Missing ⚠️
...uation/swebench_service/swebench_service/runner.py	92.30%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #438   +/-   ##
=======================================
  Coverage        ?   81.90%           
=======================================
  Files           ?      148           
  Lines           ?    19638           
  Branches        ?        0           
=======================================
  Hits            ?    16085           
  Misses          ?     3553           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nv-alicheng]

Review Council — Multi-AI Code Review

Reviewed by: Claude + Code-Quality | Depth: thorough

codex was unavailable in this environment — Claude + Code-Quality review. See the summary comment for the tiered breakdown.

[nv-alicheng]

Review Council — Multi-AI Code Review

Reviewed by: Claude + Code-Quality | Depth: thorough
(codex CLI unavailable in this environment — Claude + Code-Quality only.)

New Pyxis/SLURM SWE-bench subsystem. Subprocess calls use list-argv form with a sanitized env (safe_srun_env) — no shell=True, low command-injection risk. Both reviewers converged strongly on one theme: infrastructure failures are silently swallowed and laundered into low accuracy scores rather than propagated. 14 issues posted (26 raw findings from the two reviewers, deduped/merged).

🔴 Must Fix (high)

#	File	Line	Category	Reviewer	Summary
1	pyxis_worker.py	195	data-integrity	Both	Per-instance failures swallowed + make_run_report always writes success → an infra fault (srun fail, missing SLURM_JOB_ID, node down, registry auth) becomes a passing-shaped artifact with an undercounted score. Root cause of #2/#3.

🟡 Should Fix (medium)

#	File	Line	Category	Reviewer	Summary
2	pyxis_worker.py	138	error-handling	Both	Non-zero container exit dropped with no report/signal; no infra-vs-patch-fail distinction, no retry
3	pyxis_environment.py	162	error-handling	Both	execute launders srun TimeoutExpired/exceptions into returncode:-1 → cluster contention looks like "model's command failed"
4	pyxis_worker.py	70	design	Both	Monkeypatches upstream swebench globals with no hasattr guard — breaks silently on upstream rename
5	runner.py	747	bug	Claude	Pyxis result-path lacks Docker's rglob fallback → spurious RunnerError if upstream report filename changes
6	pyxis_environment.py	242	code-quality	Quality	__del__ bare except Exception: pass — violates repo except-comment rule
7	runner.py	761	code-quality	Quality	Function-level import w/o circular-avoidance comment (repo bans lazy imports); recurs at pyxis_environment.py:181 & pyxis_worker
8	test_runner.py	1042	testing	Claude	_evaluate_instance success/report-write branch untested (only the nonzero path is)
9	test_runner.py	764	testing	Claude	execute exception path + Submitted detection + comma/missing-repo guards untested

🔵 Consider (low)

#	File	Line	Category	Reviewer	Summary
10	pyxis_environment.py	106	security	Claude	Staging tmp dir chmod 0o1777 world-writable, bind-mounted as container /tmp on shared node
11	pyxis_environment.py	78	bug	Claude	resolve_image doesn't lowercase instance_id (Docker path does) → latent image-pull mismatch
12	runner.py	793	code-quality	Quality	Stringly-typed runtime → Literal["docker","pyxis"]
13	runner.py	706	code-quality	Quality	DRY: subset→dataset map + run-id gen duplicated across _run_eval overrides
14	runner.py	754	concurrency	Claude	Single-node enroot cleanup invariant undocumented; + README should state SLURM-allocation precondition (degraded, not errored, outside one)

Also raised, dropped as minor at this cap: magic image arch/version constants (pyxis_environment.py:87), hardcoded FQ class-name string repeated in two modules (runner.py:651, pyxis_worker.py:84), _cleanup_containers override discarding unused params (runner.py:763), **kwargs: Any erasing the typed PyxisEnvironmentConfig (pyxis_environment.py:100).

⚠️ Commit hygiene: 11 commits including 2 apparent fixups. Consider squashing before merge.

[leopck]

@hvagadia looking through this code, for the pyxis environment and worker support, as we are adding support into mini-swe agent itself, I would think it would make sense to upstream support for pyxis into mini-swe agent and open a PR in their repo instead for the pyxis support. https://github.com/SWE-agent/mini-swe-agent/tree/main/src/minisweagent/environments seems like the correct place. This would be great for the wider audience at Nvidia as well since others at Nvidia would be able to reuse your components for the pyxis support unblocking others.

Right now my biggest concern is that there is some monkey patched codes:

swebench.get_sb_environment = get_pyxis_environment
swebench.ProgressTrackingAgent = LiveTrajectoryAgent

it will cause this module to break if the API changes. If we want to adopt this into this repo, I think we must at the minimum do an assert to pin this version of mini-swe of this library support to ensure that users don't try to use other versions of mini-swe with this library e.g. (assert minisweagent.__version__) But this will incur cost of maintenance into endpoints repo and tied down to a very locked version which should be our last resort for this in the event upstream doesn't accept it.

Upstreaming this module would be the path of less maintenance. In the meantime we could sustain a fork while we wait for the upstream to accept this PR and only if the upstream maintainers do not accept, then I believe we should sustain this.

I'll let @nv-alicheng to top up any thoughts on this as well. Let me know what you think about this @hvagadia.

Btw, I'm only referring to the agent part, but the eval part of this code look justified and it does look like something that we should own. (_evaluate_instance + _EVAL_SCRIPT)

[nv-alicheng]

Agreed with @leopck - the monkeypatching seems a little hacky to me (I believe this is the Medium - 4 in my agent review post), which I think should be maybe bumped up.

Is it possible to make some child class to override and use that instead, or add it directly to swebench / make a fork?

[hvagadia]

Summary from today's discussion: I will go ahead and try to create a PR to mini-swe agent and test out the waters. Independent of that, I think we should merge this PR to unblock our experiments. If in future mini swe accepts our proposal we will update accordingly. @leopck @nv-alicheng

[leopck]

I think we should merge this PR to unblock our experiments. If in future mini swe accepts our proposal we will update accordingly

Agreed, let's unblock and get this merged. If the PR does get merged upstream, we can always remove it from the codebase.

[arekay-nv]

Thanks for this - please add some documentation especially to the pyxis path for users who might not be too familiar with pyxis. Also some choices appear arbitrary but are derived from compatibility with the docker path - please document those as well.