Contract clarification for `Buf::remaining`

[mumbleskates]

I've realized that I have possibly been assuming a stronger invariant for the return value of Buf::remaining than exists, and would appreciate clarification.

Currently the documentation states:

1. the method Returns the number of bytes between the current position and the end of the buffer
2. the return value is never less than the length of the current chunk
3. the return value must not change unless the length of the buffer is somehow modified or the buffer is advanced

One reading of this implies a strong invariant: "remaining() must always return the exact number of bytes that would be read before the end if the whole buffer were read right now".

Another reading (hopefully a wrong one! this would be very inconvenient!) could be that the apparent remaining length of the buffer may change any time it is advanced, so long as it appears to be at least as long as the current chunk, and that it need not always be exactly the number of bytes that would be read before the end immediately after the call. The documentation of Buf::advance does not specify how the return value of remaining should change after advance is called.

Can you confirm that the stronger invariant must hold by contract? Should we strengthen the language in the trait method docs, or am I jumping at shadows here?

[Darksonn]

I think the main question is whether and when the buffer is allowed to become larger. I do not think we want to allow it to become smaller out of nowhere.

[mumbleskates]

If it's allowed to "become" larger while only being used via the Buf trait, that is also a pretty big problem for me. And it does seem like a violation of the remaining() method's contract.

I'm thinking here of being a generic function, and being given an object i only know is impl Buf with no other way to influence it. It probably shouldn't be possible for that buffer to appear to change in length: Buf is an infallible trait and unsuitable for i/o, so it doesn't really seem useful or appropriate for data outside the scope of the process.

[Darksonn]

I haven't thought that much about this case. I tend to think of Buf like an Iterator of u8 like the docs suggest that we do.

The other potential possibility is buffers with a length greater than usize::MAX. Imagine a 10 GB buffer on a 32-bit machine. There's no limitation against returning the same chunk of memory twice, which can easily happen via Buf::chain(my_bytes.clone(), my_bytes.clone()).

[mumbleskates]

i guess i'm hoping for a "trusted remaining" guarantee then, analogous to TrustedLen. it holds true for all the Buf impls i've come across to date (excepting your examples of buffers comprised of oversized chains-of-clones, which saturate their remaining). Trusting this property allows having a kind of Take analog with the minimum possible overhead, where the wrapped Buf can be reborrowed freely and all that needs to be tracked is the number of bytes that should be remaining at the end.

it sounds like slightly more work is needed in the wrapper for this to work when remaining is not always trusted.

[mumbleskates]

oof, now i'm becoming even less sure again. The bytes crate itself seems to trust Buf::remaining: see BufMut::put.

For BufMut implementations that have a trustable remaining_mut, this code does one of two things when the Buf provides more bytes than it said it would:

1. if advance_mut panics, it will panic
2. if advance_mut saturates to the end of the mutable buffer and does not panic, it will get caught in an infinite loop copying into the empty self.chunk_mut()

Currently, if we assume we are copying into normal BufMut impls that represent flat allocations we can probably ignore the edge cases with saturating usize::MAX "remaining" return values, as we would already fail to allocate that many bytes. This actually does seem like a pretty strong case that we should prefer "remaining" to always be trusted when possible (when it is less than usize::MAX)