Repository navigation
Add LLVM atomic memcpy intrinsics, expose in core/std #58599
Description
Expose LLVM's "element wise atomic memory intrinsics". In particular:
- Expose
llvm.memcpy.element.unordered.atomicasatomic_element_unordered_copy_memory_nonoverlapping - Expose
llvm.memmove.element.unordered.atomicasatomic_element_unordered_copy_memory - Expose
llvm.memset.element.unordered.atomicasatomic_element_unordered_set_memory
Expose these through functions in the std::ptr module. Each function is implemented by the equivalent intrinsic or by a loop of relaxed atomic operations if the intrinsic is not available on the target platform (TODO: Given this platform-specific behavior, can this also be exposed in core::ptr?)
copy_nonoverlapping_atomic_unordered, backed byatomic_element_unordered_copy_memory_nonoverlappingcopy_atomic_unordered, backed byatomic_element_unordered_copy_memorywrite_atomic_unordered, backed byatomic_element_unordered_set_memory
Previously discussed on the internals forum here.
Folks with the authority to approve this: Let me know if this is OK to move forward; I'd like to post it to This Week in Rust's CFP.
Activity
joshlf commentedon Feb 22, 2019
@alexcrichton Do you have thoughts on this? Want to make sure I get the go-ahead before I send somebody on TWiR's CFP down a dead end.
alexcrichton commentedon Feb 22, 2019
I suspect there'll be opinions on stabilization, but seems fine to add as unstable at least to me!
joshlf commentedon Feb 22, 2019
OK cool, thanks!
tmccombs commentedon Mar 2, 2019
I'd like to take a stab at implementing this.
joshlf commentedon Mar 2, 2019
Awesome, thanks @tmccombs ! This will unblock a lot of important work :)
joshlf commentedon Mar 2, 2019
@alexcrichton On the subject of exposing in
core::ptr: Is the ban on platform-specific logic incorea general one, or just for API-visible elements? If it's the latter, then we should be able to implement the facade incore::ptras well.alexcrichton commentedon Mar 4, 2019
Currently it's largely just a "we strongly discorage platform-specific APIs" in libcore, there's already platform-specific pieces to the implementation, especially some math-related pieces on MSVC
tarcieri commentedon May 19, 2019
I saw there was some discussions around use cases on #59155 so I thought I'd bring up mine (and perhaps you can all cross-check me that this would be the proper solution).
I maintain a crate for zeroizing memory (for e.g. wiping cryptographic keys/plaintexts) which presently relies on undefined behavior - namely mixed volatile / non-volatile memory accesses (in conjunction with compiler and hardware fences):
https://github.com/iqlusioninc/crates/tree/develop/zeroize
I'm presently (ab)using
core::ptr::write_volatile(and relying on undefined LLVM behavior / implementation details) in conjunction withcore::atomic::fence.What I'd really like is an atomic
memset-style operation, i.e.llvm.memset.element.unordered.atomic, which can write repeated copies of a value atomically. The closest thing that presently exists is volatile_set_memory, however that comes with undefined behavior when volatile and non-volatile operations are mixed.RalfJung commentedon May 19, 2019
I don't know why you think that mixing volatile and non-volatile accesses is UB. I see no issue with that per se.
That said, atomic memset might still be useful.
tarcieri commentedon May 19, 2019
It is explicitly documented as UB:
https://doc.rust-lang.org/core/ptr/fn.write_volatile.html
Here is a thread on the subject of using volatile writes for zeroing sensitive memory:
https://internals.rust-lang.org/t/volatile-and-sensitive-memory/3188
RalfJung commentedon May 20, 2019
That sentences does not say "UB" or does it say "must not". The section on when this function is UB does not mention mixed accesses.
AFAIK, it mostly refers to "normal" usage of volatile, namely interaction with IO registers: you don't want to mix volatile and non-volatile accesses on those. But I do not know of any form of UB there. @rkruppe, do you?
EDIT: Opened #60972
14 remaining items
HadrienG2 commentedon Oct 12, 2019
@joshlf The cause of this particular problem is known. LLVM's optimizer is currently unable to fuse together neighboring relaxed atomic loads and stores, so a loop of AtomicU8 operations will translate into actual byte operations in hardware assembly, which is stupidly slow.
According to @comex, that's an optimizer bug, and the relaxed atomic semantics should actually allow for such code to be optimized into e.g. an efficient
rep movqor SSE2 memcpy loop on x86_64.RalfJung commentedon Oct 12, 2019
Note that relaxed vs unordered has no effect on inter-process shared memory (between distrusting parties). General consensus is (to my knowledge) still that this does and will require
volatile-- specifically, "atomic volatile", which LLVM has but Rust does not expose.joshlf commentedon Oct 12, 2019
IIUC from the discussion in rust-lang/unsafe-code-guidelines#152, normal volatile may be sufficient if combined with a freeze operation (which doesn't yet exist, of course)?
RalfJung commentedon Oct 12, 2019
That is true if we adapt LLVM's handling of read-write races. So, we would be deviating from C++ concurrency there as well. This is slightly better studied than LLVM's handling of write-write races, but that doesn't say much (as the latter is entirely unstudied to my knowledge).
steveklabnik commentedon Feb 1, 2021
Triage: I don't believe there's been any changes here.
m-ou-se commentedon Jan 7, 2022
Wg21's p1478r6 might be interesting. It proposes to add to C++ an
atomic_load_per_byte_memcpyandatomic_store_per_byte_memcpywhich, unlike the llvm intrinsics, both take a memory ordering as argument. The only valid options are Relaxed and Acquire/Release. It comes down to an unordered read followed by (or write preceded by) an atomic fence, which is exactly what you want in cases like a seqlock. We could use that design in Rust as well, to avoid the need for a new memory ordering.RalfJung commentedon Jan 7, 2022
Yeah that sounds like a great solution. :)
m-ou-se commentedon Jan 7, 2022
Cool. Time to add those intrinsics and and add this API as unstable to play around with then. :)
Not sure if I'll have time for that soon. So if anyone else has, please do! ^^
m-ou-se commentedon Jan 8, 2022
@Amanieu mentions here (#59155 (comment)):
Does that mean we could provide these
atomic_load_per_byte_memcpyandatomic_store_per_byte_memcpybut implement them without using those llvm intrinsics, and instead use volatile operations since we know llvm makes the guarantees we need on them? We'd still add a fence and take a memory ordering argument, to make sure it all fits in the existing memory model, but the implementation makes use of some extra guarantees llvm makes.RalfJung commentedon Jan 8, 2022
How do we know that?
I don't think LLVM makes guarantee on volatile accesses that race with atomic accesses.
RalfJung commentedon Jan 8, 2022
That sounds like something that is fixable in LLVM? (And if these operations are really adopted by C/C++ and used for seqlocks, they will want to do that anyway.)
joshlf commentedon Oct 14, 2022
@m-ou-se, could you clarify which intrinsics and what API surface you'd like to see added here? I'd like to post this issue on This Week in Rust's CFP section to see if we can find someone to implement it.