Skip to content

Compare GPU block-argmin objective up to FMA rounding (test only) - #155

Merged
ChrisRackauckas merged 2 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-gpu-argmin-test-fma
Oct 17, 2026
Merged

ChrisRackauckas merged 2 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-gpu-argmin-test-fma

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Ignore this PR until @ChrisRackauckas has reviewed it.

What and why

On main, the "block argmin matches brute force" test in test/gpu.jl fails on CUDA CI. It compares two numbers with ==:

  • sol.objective, which is prob.f(gbest.position, prob.p) recomputed on the host (src/solve.jl);
  • best_cost, which the GPU kernel computed for the same particle.

The kernel's sum(abs2, x) may be contracted to FMA, so the two values can differ by a few ulp even when the kernel picks the right particle. This PR changes only that test:

  • the chosen candidate is checked exactly with sol.u == best.best_position;
  • the objective is compared with isapprox(...; rtol = length(sol.u) * eps()).

Evidence

The CI gaps are all 1 ulp. GPU Tests on main has failed on every run since 2026-10-09 (593a4a2, ef0c3ad, df60a25) and was green through 2026-10-03 (956709f). The only repo change in that window is a JET compat bump. CUDA went from 6.2.2 to 6.4.1 and GPUCompiler from 1.23.0 to 2.10.0. Examples from the failing logs:

Evaluated: 0.4955911655627836 == 0.49559116556278354   (run 38819100268, main)
Evaluated: 0.740403174112652 == 0.7404031741126521     (run 38812815984)
Evaluated: 32.559608436967494 == 32.5596084369675      (run 38327786242)
Evaluated: 1.0525845108716991 == 1.0525845108716994    (run 38327786242)

A wrong candidate would almost never come out within 1 ulp of the true minimum.

FMA contraction explains a gap of this size. I ran a CPU script comparing a plain left-to-right sum(abs2, x) with an FMA-chained one, for D = 10 and 100k random points:

plain vs fma-contracted sum(abs2, x), D=10: differ in 0.31207 of cases, max 4.0 ulp

That sampled maximum is empirical, not a bound, so the tolerance is derived from the floating-point error bound instead. Summing n nonnegative squares in either order (plain or FMA-chained) gives a relative error of at most about n·u, with u = eps/2. The two results therefore differ by at most about n·eps, so the test uses rtol = length(sol.u) * eps(), which is 10eps for D = 10.

Review probe: for x = (1, y, …, y) with y = sqrt(8.5eps), the plain and FMA sums differ by 9.0eps relative. That fails rtol = 8eps() and passes rtol = 10eps:

probe rel diff = 8.999999999999838 eps; old 8eps: false; new length(x)*eps: true

All four CI pairs above are 1 ulp, well within the bound.

Local verification

PPS_TEST_GROUP=Core julia --project -e 'using Pkg; Pkg.test()' passes on CPU. The Core group runs gpu.jl with the CPU backend, including this testset:

Regression tests          |   49     49
Reinitialization tests    |    0      0
SimpleLBFGS hybrid polish |   25     25
Algorithm interface tests |    3      3
BACKEND = CPU             |   50     50

Runic and typos report nothing on test/gpu.jl.

Not verified

  • This machine has no GPU, so I could not run the CUDA job locally. The "CUDA Tests (Julia 1)" job on this PR is the real check.
  • I have not inspected the PTX to confirm that FMA contraction is the mechanism. The 1-ulp size of the gaps and the dependency bump are consistent with it.
  • I did not run the QA group, since the PR changes no source code.

Overlap with #151

#151 changes the same assertion, as part of its scalar-cost argmin work. This PR leaves out that behaviour change so main can go green on its own. Whichever PR merges second needs a trivial rebase of this hunk.

Risk assessment

  • Risk: low
  • Blast radius: one test assertion in test/gpu.jl. No source, public API or downstream impact.
  • Evidence: the CI logs show 1-ulp gaps; the CPU FMA measurement gives at most 4 ulp; the Core group passes locally on CPU.
  • Independent review: Codex gpt-6.1-sol rated it low, MERGE: Core passes; its one non-blocking note (8eps was empirical, not a bound) is addressed in 2846e8f by rtol = length(u)*eps, the n-term bound (reviews/ParallelParticleSwarms-astra/reviews/pr-155/round1/REVIEW.md on a fleet worker host).
  • Merge: auto-merge candidate once CUDA CI is green and the independent review is low risk.

🤖 Generated with Claude Code (model: claude-opus-5-5[1m])

https://claude.ai/code/session_011aK5Xvp9NpdkzKuEwtiMGd

Risk assessment (independent review, codex-cli gpt-6-astra, reviewed head 2846e8f)

  • Risk: low
  • Blast radius: One GPU/CPU testset; solver code, dependencies and public API are unchanged.
  • Evidence: Local verification and CI details below; CI pending at both checks.
  • Independent review: codex-cli 0.162.1 gpt-6-astra rated it low, medium confidence: focused change with discriminating evidence and no observed regression.
  • Merge: auto-merge candidate — technically sound, with pending CI explicitly unverified; metadata cleanup is nonblocking.

Head reviewed: 2846e8f7804f0bb2697628b444a92cef026f72c2. Recorded before review and unchanged at the final check. Initial low-risk rating was made from the diff before reading the author's risk claims.

Authorship: commit trailers identify Claude Code / claude-opus-5-5[1m], with Co-Authored-By Claude; the body also identifies Claude Code. Reviewed normally under the explicit non-Claude-reviewer exception.


🤖 Posted by an AI agent — harness: codex-cli · model: gpt-6-astra (independent reviewer; appended by Claude Code 2.1.301 · claude-opus-5-5)
Conversation: https://claude.ai/code/session_01VZjwf7Ux9YtTzwHGHMqnsq

ChrisRackauckas and others added 2 commits October 16, 2026 07:07
`sol.objective` is `f` recomputed on the host at the chosen position, while
the particles' `best_cost` comes from the GPU kernel, where multiply-adds may
be contracted to FMA. The two differ by 1 ulp on CUDA CI. Check the chosen
candidate exactly and the objective with rtol = 8eps().

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Agent-Harness: Claude Code
Agent-Model: claude-opus-5-5[1m]
Agent-Session: https://claude.ai/code/session_011aK5Xvp9NpdkzKuEwtiMGd
Each ordering of an n-term sum of nonnegative squares has relative error at
most about n*eps/2, so host and FMA-contracted kernel values differ by at most
about n*eps. The sampled 4-ulp maximum was not a bound: x = (1, y, ..., y)
with y = sqrt(8.5eps) gives a 9eps gap, which passes rtol = 10eps.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Agent-Harness: Claude Code
Agent-Model: claude-opus-5-5[1m]
Agent-Session: https://claude.ai/code/session_011aK5Xvp9NpdkzKuEwtiMGd
@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review October 17, 2026 05:15
@ChrisRackauckas
ChrisRackauckas merged commit a73c0a0 into SciML:main Oct 17, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants