Conversation
<BytesMut as BufMut>::{put, put_slice, put_bytes} were the only BufMut
methods on BytesMut without #[inline] (tokio-rs#595 inlined extend_from_slice and
noted these as the follow-up; tokio-rs#459 did the same for Vec<u8>). Without it,
and often even with LTO once reserve_inner has been folded in, every
put_slice into a BytesMut is an out-of-line call, and put_u8 (trait
default: put_slice(&[n])) additionally pays a one-byte memcpy and a
second capacity check in advance_mut. Byte-at-a-time writers such as
varint encoders are dominated by this.
Adds #[inline] to the three methods and a put_u8/put_i8 override on
BytesMut and Vec<u8> shaped like Vec::push.
benches/bytes_mut.rs (x86_64, pinned cores):
put_slice_bytes_mut 16.97 ns -> 3.18 ns (put_slice_vec: 2.8)
put_u8_bytes_mut 492 ns -> 69 ns (put_u8_vec_push: 70)
put_u8_vec 205 ns -> 70 ns
Author
|
I was a bit surprised, personally, about the I'm admittedly a bit out of my depth here, but the assembly looks like this at Graviton4 (c8g, 253 → 92 ns) pays the forwarding round-trip every byte. Granite Rapids (c8i, 66 → 66 ns) apparently predicts and forwards the same-address store+load at near-zero cost. The new assembly (with this change) is streamlined and performs well on both architectures. |
seanmonstar
enabled auto-merge (squash)
September 10, 2026 19:31
seanmonstar
disabled auto-merge
September 10, 2026 19:31
iainmcgin
added a commit
to fallintoplace/buffa
that referenced
this pull request
Sep 20, 2026
…nthropics#437) **`encode_to_bytes` was 3–4× slower than `encode_to_vec` on every benchmark shape, with or without LTO; it is now within noise of it.** Pure performance change; no behavioural or wire impact. ## Why `encode_to_bytes` built a `BytesMut::with_capacity(size)`, wrote the message through it, and froze it. `bytes` does not mark `<BytesMut as BufMut>::put_slice` `#[inline]` (it does for `Vec<u8>`), `BytesMut` has no `put_u8` override, and LLVM folds `reserve_inner` into `put_slice`, so it stays out of line even under fat LTO. Every tag and varint byte our encoders write through `put_u8` therefore became a call, a reserve check, a one-byte libc `memcpy`, and an `advance_mut` re-check; through `Vec<u8>` the same write is a compare and a store. The doc comment already said `encode_to_bytes` is "equivalent to `Bytes::from(self.encode_to_vec())`" — this makes the implementation literally that. The upstream fix is proposed separately in tokio-rs/bytes#850; this change is worthwhile regardless, since buffa cannot make its users take a newer `bytes`, and the `Vec` path is never slower. ## Change - `Message::{encode_to_bytes, try_encode_to_bytes}`, `ViewEncode::{encode_to_bytes, try_encode_to_bytes}` and the generated lazy-view inherent methods delegate to their `Vec` twins and convert with `Bytes::from`. Output bytes and the returned `Bytes` representation are unchanged: `BytesMut::freeze` on a vec-backed buffer already went through `From<Vec<u8>> for Bytes`, and with `len == capacity` (guaranteed by the two-pass size) that conversion allocates nothing. - `Rope`'s tail writes go through the inherent, inlined `BytesMut::extend_from_slice` instead of its `BufMut` impl; `to_contiguous_bytes` builds a `Vec`. - New `benchmarks/buffa/benches/encode_sink.rs` compares the sinks — `encode_to_vec` vs `encode_to_bytes`, encode into a pre-sized `Vec` vs `BytesMut`, and "frame a header then encode into your own `BytesMut`" vs "frame then `put_slice(encode_to_vec())`" — on `log_record`, `api_response`, `google_message1` and the `log_record` view. A `bench-nolto` profile and `task bench-encode-sink` run it at both the LTO bench profile and the profile a downstream `cargo build --release` gets. ## Results `encode_to_bytes`, `main` → this change, criterion median per dataset batch, one pinned core (no-LTO / fat-LTO). Development machine (Xeon 8488C): | shape | main | this change | `encode_to_vec` (reference) | |---|---|---|---| | log_record | 21.4 / 23.3 µs | 6.46 / 6.04 µs | 6.1 / 5.9 µs | | api_response | 8.31 / 8.15 µs | 2.17 / 1.97 µs | 2.0 / 1.9 µs | | google_message1 | 257 / 264 ns | 64 / 64 ns | 61 / 60 ns | | log_record (view) | 20.9 / 20.4 µs | 5.62 / 5.51 µs | 5.5 / 5.4 µs | Clean EC2 instances (Amazon Linux 2023, rustc 1.95.0): | shape | c8i.xlarge (Xeon 6975P-C) | c8g.xlarge (Graviton4) | |---|---|---| | log_record | 16.8 / 18.4 µs → 5.83 / 5.62 µs | 24.7 / 24.4 µs → 8.84 / 8.16 µs | | api_response | 6.21 / 7.01 µs → 1.98 / 1.87 µs | 8.01 / 8.44 µs → 2.94 / 2.66 µs | | google_message1 | 199 / 216 ns → 60 / 64 ns | 305 / 309 ns → 101 / 95 ns | | log_record (view) | 16.0 / 17.5 µs → 5.28 / 4.83 µs | 23.3 / 23.0 µs → 7.74 / 7.12 µs | `encode_to_vec` on the same EC2 runs: 5.4–5.6 / 8.0–8.6 µs, 1.8–2.0 / 2.6–2.8 µs, 58–63 / 89–98 ns — so `encode_to_bytes` is now within ~5 % of it on both architectures, and LTO did not rescue the old path on either. Callers that `encode` straight into their own `BytesMut` still pay the slow sink until `bytes` changes; the bench's last two rows show that encoding to a `Vec` and appending it with one `put_slice` is ~2.7× faster even counting the copy (log_record: 21.9 µs vs 8.0 µs). ## Compatibility Patch-level. No API or wire change; regenerated lazy-view code is source-compatible (method signatures unchanged) and previously generated code keeps working against the new runtime. ## Testing `cargo test -p buffa -p buffa-codegen -p buffa-test`, `cargo clippy -p buffa -p buffa-codegen --all-targets -- -D warnings`, `cargo fmt --check`. The existing `encode_to_bytes_over_limit_panics` and two-pass ledger tests cover the delegated paths; benchmarks as above. ## Follow-ups (not in this PR) - `encode_to_vec` can write through `Vec::spare_capacity_mut()` (`&mut [MaybeUninit<u8>]` implements `BufMut`) and `set_len` once, since the size is exact: measured 1.25× (varint-heavy) to 3.2× (string-heavy) over the current `Vec` path with generated code unchanged, for one line of `unsafe`. Separate PR with its own safety argument. - `Rope`'s tail as a `Vec<u8>`: a further ~1.3–1.5× on byte-heavy tails, at ≤2 small allocations per flushed segment. --------- Co-authored-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
put,put_slice, andput_bytesare the onlyBufMutmethods onBytesMutwithout#[inline]Without it every
put_sliceinto a BytesMut is an out-of-line call (even with LTO, since folding inreserve_innerbumps it past the normal instruction limit).put_u8(which defaults toput_slice(&[n])) additionally pays a one-byte memcpy and a second capacity check inadvance_mut, which hurts byte-at-a-time writers such as varint encoders.Summary
Adds
#[inline]to the three methods, and aput_u8/put_i8override onBytesMutClean-instance numbers (Amazon Linux 2023, rustc 1.95.0, one pinned core, median of 3 passes of
cargo bench --bench bytes_mut, ns/iter):put_u8_bytes_mutput_slice_bytes_mutbytes_mut_extendput_u8_vecput_u8_vec_push(baseline)put_slice_vec