Hello, I would like to get a better understanding for async Rust. I want to learn it from the bottom up. That is, I want to understand how I can create my own executor only using std (i.e. without using any external crates but only using what Nightly Rust provides).
I'm having a pretty hard time.
I do (in principle) understand Pin, though things get pretty difficult when suddenly Pin<&mut T> or Pin<Box<T, Global>> are involved, or methods like as_ref vs get_ref, or boxes and dyn objects. That's my smallest issue though (I think).
The documentation of std::future::Future makes sense to me. The only "strange" thing is the pinned pointer to the poll function, but I have assumed that's because many Futures needs to be pinned because they internally work with pointers that require the memory location to be fixed.
The std::task::Context also makes sense to me. At least regarding how to use it from within the poll function.
When it comes to writing an executor, however, I need to create such a Context, which in turn requires that I provide a std::task::Waker. The Waker is just a wrapper for RawWaker, and that one requires me to work with untyped pointers (*const ()) and RawWakerVTable.
That raises a lot of questions:
What needs to be stored in the structure pointed to by data: *const()
It is correct that I can not use closures for clone, wake, wake_by_ref, and drop, right?
Do I need to clean up memory manually (in drop)? Can I somehow "leak" an Arc and then manually drop it in the drop function using decrement_strong_count, or how does this usually work?
Is there any simple example of an executor that only uses std? Or any example for a RawWakerVTable and their corresponding functions and data structure?
The tutorials I checked always seem to resort to using external crates. Maybe that's because in past, the async support of Rust's std lib was more limited?
I do understand the motivation for not using std here (using RawWakerVTable requires unsafe code), but like I said, I want to understand things from the bottom up.
I'm sure I will get back to the Tokio tutorial afterwards (or in the middle of my process to understand everything), and likely I will be switching my current development from "normal Rust" to "async Rust", and Tokio in particular.
Thanks again for all the help. You really helped me a lot!
After finally understanding how to use RawWakerVTable (and Arc::into_raw and Arc::from_raw, which are a bit confusing as they add/substract an offset!), I figured out there may also be an easier interface in the standard library: The std::task::Wake trait, which supports conversion intoWaker.
I haven't tested it yet, but it might be an alternative to writing unsafe code or using the futures crate, as done in the Tokio tutorial. Maybe std::task::Wake is the basically the same as futures::task::ArcWake?
it might also helpful checking out the Little book I wrote on the topic of building your own simple executor. The context was a ‘no-std‘ environment for bare metal Raspberry Pi developments assuming an allocator is present.