Understanding async Rust from the bottom up (using std only)

Hello, I would like to get a better understanding for async Rust. I want to learn it from the bottom up. That is, I want to understand how I can create my own executor only using std (i.e. without using any external crates but only using what Nightly Rust provides).

I'm having a pretty hard time.

I do (in principle) understand Pin, though things get pretty difficult when suddenly Pin<&mut T> or Pin<Box<T, Global>> are involved, or methods like as_ref vs get_ref, or boxes and dyn objects. That's my smallest issue though (I think).

The documentation of std::future::Future makes sense to me. The only "strange" thing is the pinned pointer to the poll function, but I have assumed that's because many Futures needs to be pinned because they internally work with pointers that require the memory location to be fixed.

The std::task::Context also makes sense to me. At least regarding how to use it from within the poll function.

When it comes to writing an executor, however, I need to create such a Context, which in turn requires that I provide a std::task::Waker. The Waker is just a wrapper for RawWaker, and that one requires me to work with untyped pointers (*const ()) and RawWakerVTable.

That raises a lot of questions:

  • What needs to be stored in the structure pointed to by data: *const()
  • It is correct that I can not use closures for clone, wake, wake_by_ref, and drop, right?
  • Do I need to clean up memory manually (in drop)? Can I somehow "leak" an Arc and then manually drop it in the drop function using decrement_strong_count, or how does this usually work?
  • Is there any simple example of an executor that only uses std? Or any example for a RawWakerVTable and their corresponding functions and data structure?

The tutorials I checked always seem to resort to using external crates. Maybe that's because in past, the async support of Rust's std lib was more limited?

extreme is a working async runtime implementation in 44 lines of code without any dependencies besides the std.

6 Likes

Another async runtime, with no dependencies besides alloc, is described as part of the OS-building tutorial.

1 Like

Thank you both a lot. That kind of examples/explanations were what I've been looking for!

You should also check out this chapter: Async in depth | Tokio

Thanks, I will.

Actually I have seen that chapter already (when reading about Tokio). I did stop when it said:

Instead of using RawWakerVTable directly, we will use the ArcWake utility provided by the futures crate.

I do understand the motivation for not using std here (using RawWakerVTable requires unsafe code), but like I said, I want to understand things from the bottom up.

I'm sure I will get back to the Tokio tutorial afterwards (or in the middle of my process to understand everything), and likely I will be switching my current development from "normal Rust" to "async Rust", and Tokio in particular.

The source code behind the ArcWake utility can be found here.

1 Like

Thanks again for all the help. You really helped me a lot!

After finally understanding how to use RawWakerVTable (and Arc::into_raw and Arc::from_raw, which are a bit confusing as they add/substract an offset!), I figured out there may also be an easier interface in the standard library: The std::task::Wake trait, which supports conversion into Waker.

I haven't tested it yet, but it might be an alternative to writing unsafe code or using the futures crate, as done in the Tokio tutorial. Maybe std::task::Wake is the basically the same as futures::task::ArcWake?

Hi there,

it might also helpful checking out the Little book I wrote on the topic of building your own simple executor. The context was a ‘no-std‘ environment for bare metal Raspberry Pi developments assuming an allocator is present.

BR

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.