[bmehta001]

Current main integration — 41a782f28

Merged main at 0f8d01c47 without rewriting history. The native stack remains #2942 → #2956 → #2964; upper PR branches were not changed.

• Keep shared state-described checkpoint layouts, snapshot storage and prefix indexing across Bolt/Qwen; retain Qwen's new unified context-tier implementation.
• Route Bolt's upstream dynamic prefill and selective auxiliary-row gathering through checkpoint-aware planning. Large eager passes return recurrent slot 7; fixed/R8 tail handling, verification arithmetic, per-generator search.chunk_size, and target-only decode remain supported.
• Mark restored KV rows dirty so reset clears a restored prefix even when an aligned cache hit performs no target pass.
• Add actual-model dynamic-prefill regression coverage for default/R8/9/1024/4095/4096 chunk sizes, aligned and unaligned prompts, warm output parity, selected recurrent rows and reset before decoding. Keep existing fixed-R8 prefix qualification explicitly on R8.

Verified: Windows VS 2026 and Linux/WSL Release builds; 568 Windows and 571 Linux tests passed. Each broader run had two explicit skips: no registered WebGPU device, and the actual CUDA PagedAttention padding test requires an admitted GPU plus a compatible runtime. All five disabled-name cases in the selection were explicitly enabled and executed; no negative test filters were used. Repository lintrunner passes for the exact delta against current main; unchanged upstream lint findings were not rewritten.

Not yet hardware-qualified at this head: the new dynamic-checkpoint regression and actual-package numerical/performance parity. All GPUs on the reachable H100 host were occupied; the alternate host timed out. No GPU jobs or deployed services were disturbed. Checkpoint splits/chunk sizes can change prefill floating-point arithmetic, so earlier fixed-R8 hardware results below are historical evidence, not qualification of this integration.

---

Summary

Enable bounded prefix reuse for native nemotron_bolt, shared by Generators and Engine Requests using the same Model, and consolidate reusable native-prefix infrastructure for Bolt and Qwen.

• Preserve complete eight-token Bolt checkpoints, including packed dense KV, committed recurrent state and auxiliary context. Restore exact token-prefix matches and recompute unaligned tails without changing target block boundaries.
• Add the single-slot Bolt Engine adapter with authoritative-history replay, current-turn cached-token accounting and cumulative speculative metrics. Serialize shared CUDA-graph sessions and snapshot access.
• Fold the reusable direction of Share native prefix checkpoint indexing and storage policy #2948 into this implementation, without importing its older integration base: shared state-described checkpoints, snapshot policy, typed checkpoint indexing, optional publication cleanup and checked packed-buffer accounting.
• Resolve native batching settings once and explicitly configure memory budgets from that owned policy. Support device_memory_budget_bytes, device_memory_headroom_bytes, gpu_utilization_factor, snapshot tier budgets and snapshot count limits without depending on mutable configuration ordering.
• Keep backend-specific eligibility, synchronization, rollback and restore/handoff. Qwen indexes remain Engine-session-scoped; Bolt indexes remain Model-scoped.
• Document Bolt configuration, dense versus unified Qwen state, and an integration contract for enabling prefix caching in another native backend.

The existing native stack (2952) remains ordered #2937 (merged), then #2942. No stack history is rewritten.

Defaults, costs and limitations

Supported non-sliding targets default to prefix caching enabled when engine.dynamic_batching is configured and prefix_caching is omitted. This includes eligible single-slot native Bolt/Qwen configurations, even if the section was added only to request memory controls. A target sliding-attention window defaults off and requires explicit prefix_caching: true; a windowed drafter alone does not opt out the target. Existing unsupported backend/auxiliary combinations remain unsupported.

This default can allocate retained checkpoint storage and perform capture/copy work on cold prefill, even before any reuse occurs; it is not a zero-cost switch. Without an explicit snapshot storage policy, retain at most one device-resident checkpoint. Set prefix_caching: false to use native memory controls without prefix capture/retention. max_snapshots: 0 or explicitly all-zero snapshot tier budgets also disable native reuse. No native prefix cache is created without a configured batching policy section.

Existing byte budgets and count limits bound retention. Explicit RAM/disk tiering adds allocation, transfer and spill-I/O costs. Allocation/spill publication failure may refuse caching; device and restore errors still propagate. Existing demotion-failure, stale-entry cleanup, lease protection and metrics fixes are preserved. CUDA snapshots use releasable non-arena reservations so demotion also releases their physical backing.

Bolt remains a greedy, single-slot CUDA backend requiring a windowed DFlash2 drafter. Only complete eight-token boundaries are cached. Native Bolt prefill remains synchronous; external chunked-prefill scheduling is separate work. Memory controls validate fixed model/state allocations and bound snapshot VRAM; they do not dynamically resize those fixed allocations or impose a hard allocator quota.

The shared helpers are internal, not new public configuration or automatic support for arbitrary models. Paged execution retains its separate block identity and scheduling contracts.

Independent preparation for Bolt execution stack 2955

f5a3965e5 prepares narrow integration seams without importing the unmerged #2921/#2923/#2961/#2963 stack or changing this PR's base.

• Checkpoint-aware prefill planning splits target passes at the retained eight-token boundary, independently of the executor's maximum width. The current executor still uses fixed R8.
• The prefill executor returns its selected recurrent row and prediction; checkpoint bindings and prefix restoration are separate from tensor allocation and target forwarding.
• Generator and native Engine state creation share one checked typed factory. Existing eligibility, CUDA-session ownership, memory controls, cleanup and acknowledgement accounting remain intact.

Windows VS 2026 and Linux/WSL Release builds pass; all 38 focused tests pass on each platform, with no selected skips or disabled cases and no negative filters. Repository lintrunner and whitespace checks pass. The added cases cover checkpoint splitting, auxiliary-window offsets, restored prefixes, large/nonaligned planning widths and short/invalid chunks.

This is preparatory refactoring, not support or qualification for dynamic-R prefill, R1 ordinary decode or target-only Bolt. Large-width tests exercise planning only. Integrating the new executor/selected auxiliary rows, restored-KV dirty-row bookkeeping, and changed prefill numerical policies still requires the execution stack and fresh actual-model GPU qualification.

Review fixes and validation

f5cd7e1cf addresses private greedy defaults versus turn overrides, per-Engine event attribution, acknowledgement-ordered adoption metrics, physical snapshot reclamation, persistent cleanup failures and geometric index growth. Source-matched native qualification, merge-base benchmark spread and copy profiles are posted with explicit provenance and limitations.

2c9ced17a removes the remaining snapshot-cache friend through Entry lifecycle/allocation operations while keeping accounting, spill ownership and offsets private. It also fixes both macOS arm64 compilation failures by matching PrefixCacheMetrics's uint64_t member-pointer type instead of assuming it aliases size_t.

Source / validation layer	Result
2c9ced17a: Windows VS 2026 CPU build and selected regressions	448 passed, zero selected failures/skips; disabled-name benchmarks explicitly enabled
2c9ced17a: Linux/WSL CPU build and selected regressions	450 passed, zero selected failures/skips; disabled-name benchmarks explicitly enabled
2c9ced17a: fresh GitHub/Azure CI	Both previously failing macOS build lanes pass; remaining CI in progress
f5cd7e1cf: Windows CPU regressions	444 passed
f5cd7e1cf: Linux/H100 shared/native regressions	463 passed, including physical Qwen handoff and changing-size physical-memory reclamation
f5cd7e1cf: actual Bolt H100 regressions	All 5 passed, including parity/reuse, ownership teardown, resolved-default admission and cache-event attribution
f5cd7e1cf: cache-disabled merge-base comparison	Same H100/model/supported-plugin runtime; short/long TTFT and steady decode differences below 0.1%, with exact output/speculative-counter parity
Current changed C++ files	clang-format 20 and whitespace checks passed

The 912 retained f5cd7e1cf test executions had no selected skips; their 23-file source manifest was verified against that commit with checkout line endings accounted for. These results are not represented as a new GPU run of 2c9ced17a. Windows validation is CPU-only.

The Qwen regression uses actual tiny dense/head ONNX sessions and a folded validating target through supported device operations: native RAM/disk capture/restore, physical packed rows/positions/lengths, pending predictions, FastHandoff and resumed prefill/decode across compression boundaries. It does not claim full large-package separate-prefill performance qualification.

Real paged Qwen 27B qualification remains at the earlier revision; the native Flash-Next fixture and real Bolt runs are not substitutes for that workload. Earlier supported-plugin 27B results and runtime provenance remain historical evidence, not final-head requalification.

[Copilot]

🟡 Changes recommended

The change breaks compilation for supported C++17 builds by using std::erase_if and leaves shared-session teardown unsynchronized with concurrent execution.

2 open issues

• Replace C++20 std::erase_if for C++17 builds · New
• Serialize teardown with executions on shared sessions · New

What changed in this PR

Adds opt-in, bounded prefix reuse for native nemotron_bolt, shared by Generators and Engine Requests using the same Model and backed by the existing snapshot storage system.

Changes:

• Captures and restores complete eight-token checkpoints using bounded VRAM, RAM, or disk storage.
• Adds a single-slot Engine adapter with authoritative-history replay, Request-owned completion, and cache/speculative statistics.
• Adds regression coverage and documents configuration and backend limitations.

File	Description
test/​cpp/​engine/​nemotron_bolt_tests.cpp	Tests snapshot packing, retention, parity, continuation, and cancellation.
test/​cpp/​engine/​executor_result_tests.cpp	Tests cached-token accounting, EOS handling, and malformed progress.
src/​models/​nemotron_bolt.h	Declares prefix-cache structures and shared execution locking.
src/​models/​nemotron_bolt.cpp	Implements snapshot capture, restoration, and prefix-aware prefill.
src/​engine/​scheduled_requests.cpp	Validates and credits cached prompt progress before token commitment.
src/​engine/​request.cpp	Preserves Engine EOS behavior for Bolt Requests.
src/​engine/​nemotron_bolt_executor.h	Declares the native Bolt Engine dependency factory.
src/​engine/​nemotron_bolt_executor.cpp	Implements single-slot execution, history replay, and statistics.
src/​engine/​fixed_slot_scheduler.h	Exposes prefix-cache metrics through fixed-slot sessions.
src/​engine/​executor_result.h	Adds cached-prefix counts to prefill progress.
src/​engine/​backend_factory.cpp	Routes Bolt models to the native Engine adapter.
docs/​paged_attention_engine.md	Documents Bolt caching configuration, behavior, and limitations.

_{Review effort: Balanced}

[bmehta001]

Fixed Android C++17 compatibility in 30bf467: snapshot pruning now uses erase-remove instead of C++20 std::erase_if.

Validation: 126 Windows/VS 2026 regression tests passed, with no skipped tests. The original source reproduced the erase_if compiler error with the Android NDK; the corrected production translation unit passed both Android and native Linux C++17 compilation. Local Android checks used NDK r29, x86_64/API 27; the PR's full Android CI uses NDK 27.2. LLVM 20 formatting and git diff --check passed.

The push starts fresh CI; no stale-head jobs were manually rerun.

[bmehta001]

Updated in 69fdd02: merged main without rebasing/force-pushing and resolved the shared-session teardown thread. Validation on the pushed commit: Windows VS 2026 CPU build succeeded and all 127 selected Engine regressions passed; Linux CUDA/H100 build succeeded and all 130 selected regressions passed, including the new actual-model destruction-lock regression and both existing Bolt correctness/lifecycle tests. Zero failures or skipped/not-run tests in either selection; the pre-existing disabled microbenchmark was explicitly enabled. clang-format --dry-run --Werror passed for every C++ file in the PR, and whitespace checks passed. GitHub now reports MERGEABLE. Copilot rereview was requested; no new bot review has arrived yet.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[bmehta001]

Addressed all six remaining review threads from #2937 in 1eae3e4, and applied analogous feedback to Bolt:

• Document Qwen's scheduling-budget chunk remapping, DFlash context-window requirement/caps, and the explicit 8,192-token chunk used by the earlier H100 validation; warn once at native Engine initialization for a too-small effective chunk.
• State explicitly that Qwen and Bolt reject device_memory_budget_bytes; describe device headroom and alternative snapshot tiers.
• Skip unused decode KV/index/bank buffers when capturing separate dense-prefill checkpoints; retain the dense state used to reconstruct them.
• Protect cache-policy initialization ordering with a comment and single-slot configuration coverage.
• Prune/count evictions even when Qwen or Bolt publication fails. Add regressions for Qwen eviction followed by refusal or bad_alloc, and Bolt eviction followed by a real storage refusal. Remove the vacuous shorter-prompt eviction assertion.

Validation: 200 Windows VS 2026 tests and 202 WSL Linux tests passed, with zero skipped/not-run selected tests. The disabled-name Engine microbenchmark was explicitly enabled; no negative filters. Both cached offline builds succeeded. Changed-line clang-format (20.1.8) and staged/unstaged whitespace checks passed. This follow-up did not rerun real CUDA model inference.

Replied individually to the six original threads and requested a fresh Copilot review on this head.

[Copilot]

🔵 Needs a closer look

Shared CUDA-graph lifecycle and GPU-memory reclamation require human runtime qualification because reported GPU runs predate the final revision and final-head actual-model GPU verification is unavailable.

No open issues 🎉

_{Review effort: Balanced}

[bmehta001]

The matching ORT Python/core/plugin versions expose a real CUDA QSA IndexShare reuse defect: reuse intentionally holds its compressed-state prefix, but the published kernel rejects it when speculative query positions advance. The fix is in microsoft/onnxruntime#34217. On an idle A100, the new regression fails before the fix; afterward 24 ORT CUDA tests and all 114 GenAI Engine Python tests pass, including all 18 IndexShare cases, with zero skips/exclusions. GenAI validation used daa6157 and the unchanged CI core with only the rebuilt plugin substituted. The later concurrent main merge into this branch is preserved. The pinned published plugin still needs the upstream fix; rerunning unchanged MTP tests will not resolve that dependency defect.

[bmehta001]

Supersedes my earlier runtime-upgrade plan: 75a0f6c restores both GPU workflows exactly to main's dependency setup: ORT core/Python 1.31.0, CUDA NuGet plugin 0.3.0-dev.20261009143618, and Python CUDA shim 0.3.0. Linux's build dependency remains 1.26.0, as on main.

Removed only SeparateDenseSnapshotsRestoreAndHandoffAcrossCompressionBoundaries and its seven dedicated fixture files. That synthetic graph explicitly enabled folded execution and used the newer SpeculativePrefixCommit completion-ring publisher; simply removing that node would break folded decode. The remaining checkpoint implementation/tests are unchanged, and no Engine MTP or IndexShare Python tests were edited.

Validation against the actual unmodified main dependencies: VS 2026 Windows CPU build + 51 retained tests passed; Linux/A100 CUDA build + 61 retained tests passed, including all seven native Qwen tests and physical CUDA snapshot reclamation. Three optional native Bolt model tests skipped on each platform because their external model packages are unavailable. C++ formatting (LLVM 20), workflow YAML/action SHA pins, inline PowerShell parsing, and diff checks passed.

The existing IndexShare tests will again capability-skip with main's older runtime; this downgrade does not fix that independently reproduced ORT kernel bug. microsoft/onnxruntime#34217 remains valid and open, but #2942 no longer depends on that newer runtime/plugin. Fresh CI is running; not yet claiming it is green.

[Copilot]

🟡 Changes recommended

A snapshot restore failure can leave partially written KV rows uncleared because the dirty extent is not recorded before Reset() runs.

1 open issue

• Mark partially restored KV rows dirty before restore can fail · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

A critical failure for valid requests, alongside cache-compatibility, allocation-error, and turn-accounting risks, requires human assessment of the change’s impact.

No open issues 🎉

1 resolved since last review

• Mark partially restored KV rows dirty before restore can fail

_{Review effort: Max}