Conversation
<BytesMut as BufMut>::{put, put_slice, put_bytes} were the only BufMut
methods on BytesMut without #[inline] (tokio-rs#595 inlined extend_from_slice and
noted these as the follow-up; tokio-rs#459 did the same for Vec<u8>). Without it,
and often even with LTO once reserve_inner has been folded in, every
put_slice into a BytesMut is an out-of-line call, and put_u8 (trait
default: put_slice(&[n])) additionally pays a one-byte memcpy and a
second capacity check in advance_mut. Byte-at-a-time writers such as
varint encoders are dominated by this.
Adds #[inline] to the three methods and a put_u8/put_i8 override on
BytesMut and Vec<u8> shaped like Vec::push.
benches/bytes_mut.rs (x86_64, pinned cores):
put_slice_bytes_mut 16.97 ns -> 3.18 ns (put_slice_vec: 2.8)
put_u8_bytes_mut 492 ns -> 69 ns (put_u8_vec_push: 70)
put_u8_vec 205 ns -> 70 ns
Author
|
I was a bit surprised, personally, about the I'm admittedly a bit out of my depth here, but the assembly looks like this at Graviton4 (c8g, 253 → 92 ns) pays the forwarding round-trip every byte. Granite Rapids (c8i, 66 → 66 ns) apparently predicts and forwards the same-address store+load at near-zero cost. The new assembly (with this change) is streamlined and performs well on both architectures. |
seanmonstar
enabled auto-merge (squash)
September 10, 2026 19:31
seanmonstar
disabled auto-merge
September 10, 2026 19:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
put,put_slice, andput_bytesare the onlyBufMutmethods onBytesMutwithout#[inline]Without it every
put_sliceinto a BytesMut is an out-of-line call (even with LTO, since folding inreserve_innerbumps it past the normal instruction limit).put_u8(which defaults toput_slice(&[n])) additionally pays a one-byte memcpy and a second capacity check inadvance_mut, which hurts byte-at-a-time writers such as varint encoders.Summary
Adds
#[inline]to the three methods, and aput_u8/put_i8override onBytesMutClean-instance numbers (Amazon Linux 2023, rustc 1.95.0, one pinned core, median of 3 passes of
cargo bench --bench bytes_mut, ns/iter):put_u8_bytes_mutput_slice_bytes_mutbytes_mut_extendput_u8_vecput_u8_vec_push(baseline)put_slice_vec