Skip to content

Inline BytesMut's specialized BufMut methods and add put_u8 fast paths - #850

Open
rpb-ant wants to merge 1 commit into
tokio-rs:masterfrom
rpb-ant:bytesmut-inline-put
Open

rpb-ant wants to merge 1 commit into
tokio-rs:masterfrom
rpb-ant:bytesmut-inline-put

Conversation

@rpb-ant

@rpb-ant rpb-ant commented Sep 10, 2026

Copy link
Copy Markdown

Motivation

put, put_slice, and put_bytes are the only BufMut methods on BytesMut without #[inline]

Without it every put_slice into a BytesMut is an out-of-line call (even with LTO, since folding in reserve_inner bumps it past the normal instruction limit). put_u8 (which defaults to put_slice(&[n])) additionally pays a one-byte memcpy and a second capacity check in advance_mut, which hurts byte-at-a-time writers such as varint encoders.

Summary

Adds #[inline] to the three methods, and a put_u8/put_i8 override on BytesMut

Clean-instance numbers (Amazon Linux 2023, rustc 1.95.0, one pinned core, median of 3 passes of cargo bench --bench bytes_mut, ns/iter):

bench c8i.xlarge (Xeon 6975P-C) master → branch c8g.xlarge (Graviton4) master → branch
put_u8_bytes_mut 402 → 66 (6.1×) 641 → 92 (6.9×)
put_slice_bytes_mut 14.1 → 2.9 (4.8×) 12.5 → 6.0 (2.1×)
bytes_mut_extend 457 → 66 (6.9×) 613 → 92 (6.6×)
put_u8_vec 66 → 66 253 → 92 (2.7×)
put_u8_vec_push (baseline) 66 → 66 92 → 92
put_slice_vec 2.7 → 2.7 4.5 → 4.5
every other row within ±3 % within ±5 %

<BytesMut as BufMut>::{put, put_slice, put_bytes} were the only BufMut
methods on BytesMut without #[inline] (tokio-rs#595 inlined extend_from_slice and
noted these as the follow-up; tokio-rs#459 did the same for Vec<u8>). Without it,
and often even with LTO once reserve_inner has been folded in, every
put_slice into a BytesMut is an out-of-line call, and put_u8 (trait
default: put_slice(&[n])) additionally pays a one-byte memcpy and a
second capacity check in advance_mut. Byte-at-a-time writers such as
varint encoders are dominated by this.

Adds #[inline] to the three methods and a put_u8/put_i8 override on
BytesMut and Vec<u8> shaped like Vec::push.

benches/bytes_mut.rs (x86_64, pinned cores):
  put_slice_bytes_mut   16.97 ns -> 3.18 ns   (put_slice_vec: 2.8)
  put_u8_bytes_mut      492 ns   -> 69 ns     (put_u8_vec_push: 70)
  put_u8_vec            205 ns   -> 70 ns
@rpb-ant

rpb-ant commented Sep 10, 2026

Copy link
Copy Markdown
Author

I was a bit surprised, personally, about the put_u8_vec difference between c8i.xlarge (which shows no change) and c8g.xlarge (which shows a 2.7x improvement).

I'm admittedly a bit out of my depth here, but the assembly looks like this at master:

loop: cmp  [rbx], rsi          ; cap == len ?
        je   grow
        mov  rax, [rbx+8]        ; ptr
        mov  byte [rax+rsi], 0x78
        mov  rsi, [rbx+0x10]     ; RELOAD len from memory
        inc  rsi
        mov  [rbx+0x10], rsi     ; store len
        dec  ebp / jne loop

Graviton4 (c8g, 253 → 92 ns) pays the forwarding round-trip every byte. Granite Rapids (c8i, 66 → 66 ns) apparently predicts and forwards the same-address store+load at near-zero cost.

The new assembly (with this change) is streamlined and performs well on both architectures.

loop: mov  rax, [rbx+8]
        mov  byte [rax+r14], 0x78
        inc  r14                 ; len lives in r14
        mov  [rbx+0x10], r14
        ...  cmp r14, [rbx] / jne loop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant