[AdityaPandeyCN]

Runs HybridPSO's L-BFGS phase in lockstep across particles. The algorithm is unchanged from main: SimpleOptimization's projected L-BFGS with LineSearch's Strong-Wolfe search. Only the execution changes.

Each particle's _lbfgs run is cut at its objective/gradient evaluations. Every host step evaluates all particles' pending points together (f, plus the gradient as one thread per (particle, direction) with AutoForwardDiff, or per particle for other AD types). It then advances each particle through _lbfgs's own control flow to its next evaluation. Finished particles pass through unchanged. Results go into the same result/result_fx arrays as before, and the rest of solve! is as on main, including the linesearch keyword.

• New maxtime keyword (default nothing) stops the local phase after that many seconds.
• Exact only if the objective returns the same float type as its input (documented).
• LinearAlgebra moves from test extras to [deps], so dot/norm are the ones _lbfgs uses.

What _lbfgs does (SimpleOptimization 2.0.1, LineSearch 0.1.19)

• Bounds: the start is clamped into [lb, ub]. The direction is zeroed in coordinates that point out of the box at an active bound. The line search's α_max is capped at the largest step that stays in the box. The accepted point is clamped again.
• Direction and history: two-loop recursion over a circular buffer of M = threshold (s, y) pairs, scaled by s'y / y'y of the newest pair. If the direction is not a descent direction, it falls back to projected -g and restarts the history count. A pair is stored only if s'y > sqrt(eps) * max(‖s‖‖y‖, eps) and is finite; otherwise the history count resets.
• Line search (Nocedal & Wright 3.5/3.6): φ(α) = f(x + αd) and φ'(α) = ∇f(x + αd)·d, a full gradient at each trial point, with φ(0) and φ'(0) evaluated afresh. Bracketing starts at α_init, doubles up to α_max, and accepts a step blocked at α_max on Armijo alone. Zoom uses cubic interpolation kept 1% inside the bracket. maxiters and zoom_maxiters bound the two loops.
• Stopping: tol = max(max(abstol, sqrt(eps)), reltol * initial projected-gradient max-norm). It stops when the projected gradient's max-norm is at most tol, after maxiters iterations, on a non-finite f or gradient, or when the direction's max-norm is at most eps.
• Line search failure: a non-descent φ'(0), exhausting either loop, or a zero feasible step ends that particle's run at its current point.

Tests

test/lbfgs.jl checks that final positions and costs are bit-identical to main's per-thread _lbfgs kernel. Cases: Rosenbrock (unbounded, bounded, active bound), BBOB F8 in 10D (bounded and unbounded), Float64 and Float32, and AutoForwardDiff, AutoEnzyme and NoAD.

StaticArrays' dot is a @simd loop, so its rounding depends on where it is inlined. Even _lbfgs in a kernel and on the host differ in the last bits. For this test only, dot is replaced by the same sum without @simd, and the replacement is removed afterwards.

The bounded-Rosenbrock and maxtime tests are kept.

The block argmin matches brute force CUDA failures also happen on main.

[ChrisRackauckas]

Why not just put the SimpleOptimization call in a kernel call?