Repository navigation
Conversation
…ication Reuse token-prefix indexing, residency pruning, and snapshot storage policy across Qwen and Bolt while preserving their state layouts, checkpoint boundaries, leases, and cache scopes. Validation: 174 Windows VS 2026 tests and 176 WSL tests pass with no skips; C++17 index probe, changed-file clang-format, and whitespace checks pass. Real-model CUDA checks were not rerun. Files changed: - docs/paged_attention_engine.md - src/engine/prefix-checkpoint-index.h - src/engine/prefix_snapshot_cache.cpp - src/engine/prefix_snapshot_cache.h - src/engine/qwen38_graph_executor.cpp - src/engine/qwen38_graph_executor.h - src/models/nemotron_bolt.cpp - src/models/nemotron_bolt.h - src/models/qwen38_graph.cpp - test/cpp/engine/prefix-checkpoint-index-tests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99b6fd2e-5a8b-47d1-916d-abe7c3ded670
bmehta001
added a commit
that referenced
this pull request
Nov 12, 2026
Fold the reusable direction of #2948 into the current #2942 implementation without importing its older integration base or losing recent budget, publication-failure, demotion, and metrics fixes. Share owned snapshot policy, typed prefix matching, optional publication cleanup, and checked packed-buffer accounting. Keep synchronization, rollback, safe boundaries, restore/handoff, and index ownership backend-specific. Configure native memory budgets explicitly from the same owned settings. Move generic policy tests into one shared suite while retaining Qwen wiring coverage. Add tiny real dense/head ONNX fixtures to verify RAM/disk restore, compression boundaries, physical packed state, resumed prefill, and decode. Document the integration contract and the white-box fixture's scope. Files changed: - docs/paged_attention_engine.md - src/engine/native-device-memory-budget.cpp - src/engine/native-device-memory-budget.h - src/engine/prefix-checkpoint-index.h - src/engine/prefix_snapshot_cache.cpp - src/engine/prefix_snapshot_cache.h - src/engine/qwen38_graph_executor.cpp - src/engine/qwen38_graph_executor.h - src/models/nemotron_bolt.cpp - src/models/nemotron_bolt.h - src/models/qwen38_graph.cpp - src/models/qwen38_graph.h - test/cpp/engine/prefix-checkpoint-index-tests.cpp - test/cpp/engine/qwen38_graph_native_tests.cpp - test/cpp/engine/qwen38_prefix_snapshot_tests.cpp - test/models/engine/synthetic-qwen38/decoder-handoff.onnx - test/models/engine/synthetic-qwen38/dense-prefill.onnx - test/models/engine/synthetic-qwen38/generate-dense-handoff.py - test/models/engine/synthetic-qwen38/qsa.data - test/models/engine/synthetic-qwen38/qsa.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06cfce86-6a11-4cf7-82b7-297f91a0b519
Contributor
Author
|
Superseded by #2942 at 8c5aac8. Its current implementation incorporates the shared native snapshot policy, typed checkpoint index/publication cleanup and packed-buffer accounting, plus a new-backend integration recipe, while preserving the later budget, demotion, failure-path and metrics fixes. Model-specific synchronization/rollback and Qwen Engine-session versus Bolt Model ownership remain explicit rather than using a generic Store callback. Windows/WSL and H100 validation, including dense restore/handoff and actual Bolt/Qwen probes, are documented on #2942. Closing this draft rather than merging its older integration base; branch and worktree are retained. |
bmehta001
added a commit
that referenced
this pull request
Nov 14, 2026
## Current main integration — `41a782f28` Merged `main` at `0f8d01c47` without rewriting history. The native stack remains #2942 → #2956 → #2964; upper PR branches were not changed. - Keep shared state-described checkpoint layouts, snapshot storage and prefix indexing across Bolt/Qwen; retain Qwen's new unified context-tier implementation. - Route Bolt's upstream dynamic prefill and selective auxiliary-row gathering through checkpoint-aware planning. Large eager passes return recurrent slot 7; fixed/R8 tail handling, verification arithmetic, per-generator `search.chunk_size`, and target-only decode remain supported. - Mark restored KV rows dirty so reset clears a restored prefix even when an aligned cache hit performs no target pass. - Add actual-model dynamic-prefill regression coverage for default/R8/9/1024/4095/4096 chunk sizes, aligned and unaligned prompts, warm output parity, selected recurrent rows and reset before decoding. Keep existing fixed-R8 prefix qualification explicitly on R8. **Verified:** Windows VS 2026 and Linux/WSL Release builds; 568 Windows and 571 Linux tests passed. Each broader run had two explicit skips: no registered WebGPU device, and the actual CUDA PagedAttention padding test requires an admitted GPU plus a compatible runtime. All five disabled-name cases in the selection were explicitly enabled and executed; no negative test filters were used. Repository lintrunner passes for the exact delta against current main; unchanged upstream lint findings were not rewritten. **Not yet hardware-qualified at this head:** the new dynamic-checkpoint regression and actual-package numerical/performance parity. All GPUs on the reachable H100 host were occupied; the alternate host timed out. No GPU jobs or deployed services were disturbed. Checkpoint splits/chunk sizes can change prefill floating-point arithmetic, so earlier fixed-R8 hardware results below are historical evidence, not qualification of this integration. --- ## Summary Enable bounded prefix reuse for native `nemotron_bolt`, shared by Generators and Engine Requests using the same Model, and consolidate reusable native-prefix infrastructure for Bolt and Qwen. - Preserve complete eight-token Bolt checkpoints, including packed dense KV, committed recurrent state and auxiliary context. Restore exact token-prefix matches and recompute unaligned tails without changing target block boundaries. - Add the single-slot Bolt Engine adapter with authoritative-history replay, current-turn cached-token accounting and cumulative speculative metrics. Serialize shared CUDA-graph sessions and snapshot access. - Fold the reusable direction of #2948 into this implementation, without importing its older integration base: shared state-described checkpoints, snapshot policy, typed checkpoint indexing, optional publication cleanup and checked packed-buffer accounting. - Resolve native batching settings once and explicitly configure memory budgets from that owned policy. Support `device_memory_budget_bytes`, `device_memory_headroom_bytes`, `gpu_utilization_factor`, snapshot tier budgets and snapshot count limits without depending on mutable configuration ordering. - Keep backend-specific eligibility, synchronization, rollback and restore/handoff. Qwen indexes remain Engine-session-scoped; Bolt indexes remain Model-scoped. - Document Bolt configuration, dense versus unified Qwen state, and an integration contract for enabling prefix caching in another native backend. The existing native stack (2952) remains ordered #2937 (merged), then #2942. No stack history is rewritten. ## Defaults, costs and limitations **Supported non-sliding targets default to prefix caching enabled when `engine.dynamic_batching` is configured and `prefix_caching` is omitted.** This includes eligible single-slot native Bolt/Qwen configurations, even if the section was added only to request memory controls. A target sliding-attention window defaults off and requires explicit `prefix_caching: true`; a windowed drafter alone does not opt out the target. Existing unsupported backend/auxiliary combinations remain unsupported. This default can allocate retained checkpoint storage and perform capture/copy work on cold prefill, even before any reuse occurs; it is not a zero-cost switch. Without an explicit snapshot storage policy, retain at most one device-resident checkpoint. Set `prefix_caching: false` to use native memory controls without prefix capture/retention. `max_snapshots: 0` or explicitly all-zero snapshot tier budgets also disable native reuse. No native prefix cache is created without a configured batching policy section. Existing byte budgets and count limits bound retention. Explicit RAM/disk tiering adds allocation, transfer and spill-I/O costs. Allocation/spill publication failure may refuse caching; device and restore errors still propagate. Existing demotion-failure, stale-entry cleanup, lease protection and metrics fixes are preserved. CUDA snapshots use releasable non-arena reservations so demotion also releases their physical backing. Bolt remains a greedy, single-slot CUDA backend requiring a windowed DFlash2 drafter. Only complete eight-token boundaries are cached. Native Bolt prefill remains synchronous; external chunked-prefill scheduling is separate work. Memory controls validate fixed model/state allocations and bound snapshot VRAM; they do not dynamically resize those fixed allocations or impose a hard allocator quota. The shared helpers are internal, not new public configuration or automatic support for arbitrary models. Paged execution retains its separate block identity and scheduling contracts. ## Independent preparation for Bolt execution stack 2955 `f5a3965e5` prepares narrow integration seams without importing the unmerged #2921/#2923/#2961/#2963 stack or changing this PR's base. - Checkpoint-aware prefill planning splits target passes at the retained eight-token boundary, independently of the executor's maximum width. The current executor still uses fixed R8. - The prefill executor returns its selected recurrent row and prediction; checkpoint bindings and prefix restoration are separate from tensor allocation and target forwarding. - Generator and native Engine state creation share one checked typed factory. Existing eligibility, CUDA-session ownership, memory controls, cleanup and acknowledgement accounting remain intact. Windows VS 2026 and Linux/WSL Release builds pass; all 38 focused tests pass on each platform, with no selected skips or disabled cases and no negative filters. Repository lintrunner and whitespace checks pass. The added cases cover checkpoint splitting, auxiliary-window offsets, restored prefixes, large/nonaligned planning widths and short/invalid chunks. This is preparatory refactoring, not support or qualification for dynamic-R prefill, R1 ordinary decode or target-only Bolt. Large-width tests exercise planning only. Integrating the new executor/selected auxiliary rows, restored-KV dirty-row bookkeeping, and changed prefill numerical policies still requires the execution stack and fresh actual-model GPU qualification. ## Review fixes and validation `f5cd7e1cf` addresses private greedy defaults versus turn overrides, per-Engine event attribution, acknowledgement-ordered adoption metrics, physical snapshot reclamation, persistent cleanup failures and geometric index growth. [Source-matched native qualification, merge-base benchmark spread and copy profiles](#2942 (comment)) are posted with explicit provenance and limitations. `2c9ced17a` removes the remaining snapshot-cache `friend` through Entry lifecycle/allocation operations while keeping accounting, spill ownership and offsets private. It also fixes both macOS arm64 compilation failures by matching `PrefixCacheMetrics`'s `uint64_t` member-pointer type instead of assuming it aliases `size_t`. | Source / validation layer | Result | |---|---| | `2c9ced17a`: Windows VS 2026 CPU build and selected regressions | 448 passed, zero selected failures/skips; disabled-name benchmarks explicitly enabled | | `2c9ced17a`: Linux/WSL CPU build and selected regressions | 450 passed, zero selected failures/skips; disabled-name benchmarks explicitly enabled | | `2c9ced17a`: fresh GitHub/Azure CI | Both previously failing macOS build lanes pass; remaining CI in progress | | `f5cd7e1cf`: Windows CPU regressions | 444 passed | | `f5cd7e1cf`: Linux/H100 shared/native regressions | 463 passed, including physical Qwen handoff and changing-size physical-memory reclamation | | `f5cd7e1cf`: actual Bolt H100 regressions | All 5 passed, including parity/reuse, ownership teardown, resolved-default admission and cache-event attribution | | `f5cd7e1cf`: cache-disabled merge-base comparison | Same H100/model/supported-plugin runtime; short/long TTFT and steady decode differences below 0.1%, with exact output/speculative-counter parity | | Current changed C++ files | clang-format 20 and whitespace checks passed | The 912 retained `f5cd7e1cf` test executions had no selected skips; their 23-file source manifest was verified against that commit with checkout line endings accounted for. These results are not represented as a new GPU run of `2c9ced17a`. Windows validation is CPU-only. The Qwen regression uses actual tiny dense/head ONNX sessions and a folded validating target through supported device operations: native RAM/disk capture/restore, physical packed rows/positions/lengths, pending predictions, FastHandoff and resumed prefill/decode across compression boundaries. It does not claim full large-package separate-prefill performance qualification. Real paged Qwen 27B qualification remains at the earlier revision; the native Flash-Next fixture and real Bolt runs are not substitutes for that workload. [Earlier supported-plugin 27B results and runtime provenance](#2942 (comment)) remain historical evidence, not final-head requalification. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Copilot-Session: fbd11940-c14d-46c7-a724-3b2c65a6cf76 Copilot-Session: 996a8a25-fe6d-45c2-8b19-c38c57c9b7e9 Copilot-Session: 6b04e9f6-4182-4bfc-bf43-e53068c42638 Copilot-Session: d81bd60e-20e8-433a-8458-000e64fdd612 Copilot-Session: b3ab443c-d6fb-4693-b5f8-d18385dbbe6d Copilot-Session: e311d5a9-0f8b-4d24-b8cf-903dba231d73 Copilot-Session: 06cfce86-6a11-4cf7-82b7-297f91a0b519 Copilot-Session: bb46f334-5418-47f2-92ef-4553146435dd Copilot-Session: 80a80d1c-8d3a-442a-aaee-96bf9e06156e Copilot-Session: 07f2040e-6d5e-4254-b25a-eb78c493427f Copilot-Session: 276e922f-3816-4c82-89a0-0702f676c388 Copilot-Session: ac28fda8-704a-4048-874c-cf80298541a0 Copilot-Session: b730396c-0065-4e06-bac7-70e6eb962060 Copilot-Session: 71f338fc-57ae-4336-8b1a-1ccf6d2ff1f7 Copilot-Session: aa4e0c1f-b9cc-448f-b6ad-77ba4d709f35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Start making native prefix caching reusable across model families by sharing checkpoint identity/indexing and storage-policy resolution between Qwen and Bolt.
PrefixCheckpointIndexwhose entries provide token identity and live residency. Share longest complete token-prefix matching, stale-entry pruning, and pre-publication capacity reservation without new virtual dispatch or payload allocations.CreateNativePrefixSnapshotCacheso both backends use the same explicit opt-in, snapshot-count defaults, and VRAM/RAM/disk budget policy.No public API/configuration changes, new synchronization, or additional GPU copies are introduced. This is the first shared infrastructure slice, not a claim that arbitrary model state can now be cached automatically.
Dependencies and temporary base
This draft deliberately targets
integration/native-prefix-cache, notmain. That branch contains:d57715015b0cbdad00e1ebd024090d9661d2c816.1f7c6d85a31f68e8f7688237b0a3f1fac10e71ff.7a97274a4e5ffe1e86888931439250511ade9d3f.The integration base preserves the malformed-backend-progress regression when resolving the shared test conflict and removes duplicate metrics-hook definitions introduced by Git's automatic merge. The base tip is
3da68bb1c2735e2ca8daa1d435557a6ea113e48a.Neither prerequisite PR was modified, merged, or restacked. The comparison shows only this refactoring layer. Do not land the temporary integration branch; after both prerequisites land, merge updated
maininto this branch, resolve any differences, and retarget this PR tomain.Validation
Fresh offline CPU builds succeeded with Windows VS 2026 and WSL/Linux GCC 13.
USE_CXX17clang-format --dry-run --Werroron every changed C++ fileFocused selection:
The selected disabled-name microbenchmark was explicitly enabled. XML results confirm zero failures, errors, skipped tests, or not-run selected tests. The explicit VRAM policy case uses the existing CUDA device double; these are CPU validation runs, not physical-GPU storage checks.
Real-model CUDA parity/latency checks were not rerun for this slice, and no performance improvement is claimed.
Follow-up direction
Use this shared foundation to introduce reusable state/range descriptions and lifecycle hooks where the existing backends demonstrate a common contract. Safe checkpoint boundaries and complete resumable state must remain explicit; this PR does not unify paged and native execution, eliminate Engine adapters, or permit cross-model checkpoint reuse.