Tracking Issue for `waker_getters`

[oxalica]

Feature gate: #![feature(waker_getters)]

This is a tracking issue for getters of data and vtable pointers for RawWaker, and the method to get RawWaker from Waker.

Public API

// core::task
impl RawWaker {
    pub fn data(&self) -> *const ();
    pub fn vtable(&self) -> &'static RawWakerVTable;
}
impl Waker {
    pub fn as_raw(&self) -> &RawWaker;
}

Steps / History

• Implementation: Implement RawWaker and Waker getters for underlying pointers #91828
• Final comment period (FCP)
• Stabilize waker_getters #129919

Unresolved Questions

• Do we also need fn into_raw(self) -> RawWaker for Waker? Feature: Provide getters of data and vtable pointer for RawWaker? #87021 (comment)

[oxalica]

Maybe we could stabilize this? Note sure what I could do next. @rust-lang/libs-api

[Amanieu]

I am concerned about as_raw, it's not clear to me how it is intended to be used. It takes Waker by reference and returns a RawWaker, which you can then decompose into a vtable and data pointer. However I don't see how you can then re-construct a reference to the original Waker: doing so with from_raw will create a second Waker which will call the drop function in the vtable when dropped. This can lead to a double-drop once the original Waker is dropped as well.

[oxalica]

@Amanieu

I am concerned about as_raw, it's not clear to me how it is intended to be used. It takes Waker by reference and returns a RawWaker

It returns &RawWaker, not RawWaker, thus it's a getter rather than an ownership transfer. Its purpose is to allow getting or checking the underlying data. Reconstructing a [Raw]Waker from the result pointers is indeed incorrect, and unnecessary due to the existence of Waker::clone.
If we do want reconstruction, I'd consider adding another fn into_raw(Waker) -> RawWaker, which also consists with Waker::from_raw. But I think it's less useful, because we can only access &mut Context and get a &Waker. We don't own Waker without explicitly cloning it somewhere.

[Amanieu]

Then I'm afraid that I don't understand how as_raw is intended to be used. What can you actually do with the returned vtable and data? I can understand passing it through FFI, but then you eventually have to pass it back to Rust code to use it as a Waker.

[oxalica]

@Amanieu

Then I'm afraid that I don't understand how as_raw is intended to be used.

It's effectively downcast_ref for Waker. It allows validating Waker VTables, ad-hoc optimization, or retrieving data from known Wakers, where the async runtime only supports Wakers from itself. I found an example in,
https://github.com/embassy-rs/embassy/blob/be55d2a39eb399d693d11f37dfa4b87cd2bd3c7a/embassy-executor/src/raw/waker.rs#L42

I can understand passing it through FFI, but then you eventually have to pass it back to Rust code to use it as a Waker.

In case of passing through FFI, we need do &Waker -> [*const (); 2] -> &Waker ¹ without ownership transfer. The last step can be done via reconstructing ManuallyDrop<Waker> and using the reference to inner. If we want to transfer the ownership via, to say fn into_raw(Waker) -> RawWaker, we need an extra Waker::clone during every poll.

Note that the ManuallyDrop<Waker> - &Waker pattern above can be found in,
https://github.com/tokio-rs/tokio/blob/c693ccd210c7a318957b0701f4a9632b2d545d6a/tokio/src/util/wake.rs#L18-L22

Footnotes

1. Since the VTable representation is repr(Rust). Here we actually need to create a new repr(C) VTable for forwarding method calls, rather than to pass the pointer through FFI directly. This is doable currently. ↩

[Amanieu]

I feel that a better story is needed for getting a &Waker from a &RawWaker. Perhaps something like this?

impl Waker {
    unsafe fn with_raw<T>(raw: &RawWaker, f: impl FnOnce(&Waker) -> T) -> T;
}

Also I do think that we should add into_raw for consistency with from_raw.

[oxalica]

I feel that a better story is needed for getting a &Waker from a &RawWaker. Perhaps something like this?

impl Waker {
    unsafe fn with_raw<T>(raw: &RawWaker, f: impl FnOnce(&Waker) -> T) -> T;
}

This API seems to be the only choice since we cannot let temporary Waker instance escape, but it would be less flexible for the tokio case. It's not possible to return a impl Deref<Target = Waker> via this API. For the FFI case, it seems to work though.

Also this API is more like a helper, which is able to be implemented outside std (RawWaker::data cannot, due to repr(Rust)). Should it also be a part of waker_getters?

[Amanieu]

I think it should be inlcuded as the counterpart to Waker::as_raw, otherwise it isn't clear how the RawWaker returned by as_raw can be safely used. This is important for the doc examples for these functions.

For tokio, it seems to only be used here. It should be easy to refactor that to use with_raw.

[oxalica]

@Amanieu
Noticing that Waker is repr(transparent), it should be safe to transmute from &RawWaker to &Waker I think?
Then the with_raw you suggested can be replaced by,

impl Waker {
    pub unsafe fn from_raw_ref(raw: &RawWaker) -> &Waker { transmute(raw) }
}

[wishawa]

I have a pretty simple use case for this: checking if two Wakers are the same.

The Future contract requires wake to be called on the Waker from the latest poll, so on every poll a Future that keeps its Waker stored have to replace its stored Waker with one newly cloned from the Context. Waker::as_raw would allow us to first compare the two Wakers, avoiding replacement (and the atomic operations involved) if they are the same.

Didn't realize will_wake does this already! Thanks @Thomasdezeeuw

[Amanieu]

Because Waker has private fields, the #[repr(transparent)] is an implementation detail and not an API guarantee that users can rely on. It isn't shown in rustdoc for this reason.

The problem with from_raw_ref is that it effectively turns this into a public guarantee: it might be something we can commit to but it would have a higher barrier for acceptance.

[Thomasdezeeuw]

To add another use case is storing the waker atomically. You can split the data and vtable into two AtomicU64 and store those atomically separately (with some careful programming) or in a single AtomicU128 (on 64 bit archs at least).

I have a pretty simple use case for this: checking if two Wakers are the same.

The Future contract requires wake to be called on the Waker from the latest poll, so on every poll a Future that keeps its Waker stored have to replace its stored Waker with one newly cloned from the Context. Waker::as_raw would allow us to first compare the two Wakers, avoiding replacement (and the atomic operations involved) if they are the same.

@wishawa wote that task::Waker::will_wake exists for this.

[conradludgate]

I think it might be natural to think that a Waker could do more than just wake(). Like how a Box<dyn Error> might be a specific kind of error with some fields we would care about.

I can imagine as situation where you might want to check if the current Waker is a FooWaker in order to extract a field from it, eg a TaskID.
(I am exploring an idea at the moment to hack in a go like context.Context typemap into core::task::Context through this mechanism.)

In order to do this, I could verify that waker.raw_waker().vtable() == &FooWakerVTable.

(in a similar respect to Error and source, it would be interesting to have a waker chain, especially with the recent push to structured concurrency with many nested sub-executors, but that is for a separate issue and another time)