Poll: Async/Await, let's talk about executors!

community
1

Async/Await is stabilizing in Rust 1.39. That means that the async ecosystem is growing rapidly. I would like to question how we deal with choosing executors when we need to spawn futures, especially in libraries that provide an async API.

I think we should allow library code to spawn futures without:

  1. bloating the dependency graph of client code with runtimes (reactors/timers/network dependencies, ...), and especially multiple versions of those if client code depends on several async libraries.
  2. deciding what executor implementation to use, and whether to spawn on a threadpool or on the current thread.
  3. limiting the use of the library to multithreaded systems (notably, WASM is currently single threaded) if the library does not otherwise require threads.

It's worth quickly touching on why should you spawn futures? Futures can be awaited or returned to the client code. While that is true, it doesn't always work out:

  • you might have to bridge not async API's to async. It might be synchronous API's, callback based API's (Web API in WASM), you might be on a single threaded environment, ... and sometimes you just need something to run concurrently and it might be part of an implementation detail and not your API.
  • you might not be in async context because you are implementing poll functions of Sinks, Streams, AsyncRead, ... In this context you cannot await, and you cannot return a future because the signature of these traits doesn't return futures.

So if we do need to spawn futures in library code, how can it be done?

Two possible designs come to mind when trying to solve the initial problem.

  1. library code that needs to spawn must always take in T: Executor, but what is this Executor trait? It turns out the trait already exists, in a library almost all async code imports: T: futures::task::Spawn. Oh, awesome. But, wait, none of the three main executor implementations we currently have: tokio, juliex and async-std implements this trait. Oops. Async-std doesn't even expose an executor object.
  2. We somehow make a global spawn function available everywhere. As an extra convenience, this does not clutter up the interfaces of your API. The downside is that it is not obvious that some piece of code spawns.

The poll below ask you whether the recommended way in Rust should be:

  1. executors should implement a Spawn trait and libraries should take a generic executor object
  2. a library that provides a global spawn function that library code can call without pulling in any executor specific dependencies and leaving decisions to the app developer

Both approaches do not need to be exclusive. Even if all executors would implement Spawn and SpawnLocal where appropriate, we could still chose to provide API's that don't require passing around executors all over the place.

The question can be seen in the light of other similar questions. In OOP, it is considered good practice to require all your dependencies in the constructor. Makes it obvious up front. The downside is cluttering up all functions with a bunch of parameters because you have to pass around stuff all over the place.

At risk of influencing the poll I would still like to point out that Rust already has a similar example: logging. The log crate makes logging available everywhere and lets app devs decide what implementation to use, where as slog requires you to pass around a logger object, and as a benefit gives extra features like structured logging.

Which should be recommended?

  • Pass around E: Spawn or E: SpawnLocal, or other traits yet to be created.
  • Make a library that exposes a global spawn function and abstracts out over implementations?
  • Both should exist and are equally valid
  • I have another solution in mind (please leave a comment and enlighten us)

0 voters

11 Likes
2

Looks like the poll above is Poll:NotReady ...

9 Likes
3

There's a complication here unfortunately: the spawn method of the Spawn trait was intended to take an unsized parameter, which is possible in nightly with the unsized_locals feature, but not in stable. Using a type parameter for the spawn method would make the trait object-unsafe I believe.

To standardize a Spawn trait, I believe we'll also need to push the unsized_locals feature--or, at the very least, part of it--through the stabilization process to release in 1.39.

The futures::task::Spawn trait gets around this be including an extension trait with a spawn method that takes a Future type parameter. This creates an unfortunate deluge of extension traits though.

1 Like
4

Thanks for pointing that out. I didn't look at the implementation of those traits enough, given that I haven't written an executor myself.

It looks like Spawn works with a FutureObject. I don't see any unstable features being used. However the SpawnExt trait actually boxes the future (for object safety I imagine) which is unfortunate.

Object safety is nice, but in itself not required to be able to take an T: Spawn.

5

I feel like passing E: Spawn or E: SpawnLocal is not only more modular, but also more explicit. It tells programmers that piece of code might spawn new tasks. We should aim to make side effects explicit and interacting with a runtime is clearly a side effect.

That's the reason I dislike functions runtime::spawn() or tokio::spawn(). They actively encourage relying on a static executor and hide the runtime dependency. They make mocking or using a different runtime for a subset of functions more difficult.

Now as @lachlansneff pointed out the Spawn traits may not be in their final form, but in the mean time I use those provided by the futures crate.

Finally, I find it's not usually very difficult to pass executors around using a service approach where some structs are going to be "services" able to spawn tasks when their methods are called (and other services). They are usually created only once and the wiring is pretty straightforward.

11 Likes
6

Thanks for the feedback.

What's remarkable about this poll is most people seem to agree with you, which doesn't correspond to the main interfaces we have available today, and which doesn't seem to correspond the popularity of the log crate vs slog either. Quite interesting.

I made a mistake in my last post. I was hoping that there were executors that don't box on spawn, but thinking about how an executor could be implemented, I quickly realized that you can't do it without boxing, since you have to store futures which are variable in size and type. Even things like join_all box, so it seems that the only path that avoids heap allocation is await.

Ah, it seems the join_1-join_5 family don't box.

ps: I'm not very clear about which traits are best right now, but tokio also provides Executor/TypedExecutor. I'm still familiarizing with their executor code. However the only way to use their executors is by using the rt-full feature with their runtime, which doesn't only pull in reactor/timer that you might not need, but also network code like tcp/udp, and mio...:frowning:

7

Interesting comparison with logging. At the end of the day what should be statically callable as a part of the runtime vs what should be explicitly passed is somewhat subjective.

My compass there is whether we're talking about application vs library.

I don't want a library to make static calls to runtime or logging because that links me with dependencies I might want to customize or upgrade. If a library crate is to log, I appreciate if it has an explicit L: Log parameter on its calls. However in my experience most crates do not log and instead return Result and let users deal with errors and logs. Maybe I'm wrong though.

Likewise, I want to be able to wire a 3rd party crate with the executor I like and appreciate using a Spawn trait bound there.

In an application, I find using the Spawn trait helpful to infer architecture, but I'd find using a Log trait cumbersome because it would have to be injected everywhere.

8

I suspect you might be misunderstanding how Futures work in Rust.

They do not work like Promises in JS: you're not supposed to immediately spawn them.

Instead, you're supposed to build up a single large Future using .await, or the FutureExt combinators (map, join, then, etc.), and then at the very top level of your application (in main) you spawn it.

If you need to convert an API (like callbacks or Promises) into a Future, you should be using futures::channel::oneshot. You don't need to spawn Futures in order to convert APIs.

If you need concurrency, spawning Futures is a pretty bad way to do that.

Instead, you should be using APIs like join, try_join, join_all, FuturesUnordered, etc.

Those APIs do not require spawning Futures, they are faster than spawning Futures, they work concurrently even in single-threaded systems (like Wasm), and they work with every runtime and every executor.

I would say that libraries spawning Futures is a massive code smell. There's very few legitimate reasons for a library to do that.

Libraries should just return Futures or Streams, and let the application handle the spawning.

I would go even further and say that the application should only call spawn once, in main. This is how runtime works. If you're spawning multiple times, you're doing it wrong.

But in the very rare case where a library does need to spawn Futures, I think it should take a T: Executor or T: Spawn or whatever bound.

3 Likes
9

Thanks for the feedback.

I agree with you that this is the ideal model and I have started to eliminate some spawns, (like creating a custom future that can be polled from within a poll method) rather than spawning it.

This post that @matklad linked from their blog was very interesting as well, about keeping the flow linear:

https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/

However it's hard to anticipate if that works in all use cases. One of the situations I found myself reaching for spawn is in constructors. An object responsible for a network connection. The code that listens for incoming connections is long lived.

Sure, I could change that, so the constructor returns (object, future) and it's up to the user to spawn that at some point, and maybe that's what I'll do, but it does make API's more complicated. It's tempting to just spawn it there and then and just return the object.

Another issue I wonder about is that the generator accumulates all the call stacks. I wonder how this scales? Maybe if you have a lot of nested async fn calls, at some point it's better to spawn a part rather than to keep accumulating in order to have but a single task in a (potentially big) application.

I guess I'll try for now to eliminate spawn from my libraries, but I will have to see how wieldy it is in a real life application which might have many task that need to run concurrently.

Let's just note this is confusing for beginners, as none of the executor libraries that provide spawn functions document this, and spawn has some attractive simplicity to it. You need that piece of code to run concurrently, you spawn it and it works.

As an example, runtime which you mention provides a global spawn function, but does not advise in it's documentation when/how to use it.

I just had a look at the async book, the chapter about spawning isn't written yet, but I hope it will advise people to avoid it where appropriate.

1 Like
10

This is another example:

https://github.com/najamelan/ws_stream_wasm/blob/master/src/ws_io.rs#L107

Where as for a network connection, it's explainable in documentation that it has an incoming and an outgoing part that need to run concurrently, in this case, the fact that the stream has to wake up pending tasks when the connection closes seems really like an implementation detail.

It feels strange to return that from a constructor, saying, here is your object, and a bunch of little tasks that (for implementation details) we need you to run concurrently for us.

11

I think the trio way of solving vis is not to return the future, it is pass a nursery as an argument.

12

Would it be safe to say that in rust the equivalent is to pass an executor in?

Actually, just thinking if an executor is passed in and used to spawn, we still don't preserve the linear flow of the nursery, which joins all tasks...

13

I don't know how your code works, but I can see many solutions:

  • The Foo::new function can return an impl Future<Output = Foo>

  • You can implement Future directly for Foo

  • For a network connection you probably want to implement Stream for Foo

  • It can use futures::oneshot::channel internally, as I mentioned, which is idiomatic

I doubt that, everything is efficiently combined into a compact struct.

But if you ever did need to split it, you can do that by boxing the Futures yourself, or using things like FuturesUnordered.

I completely agree, Rust Futures are unique, they work quite differently from how Futures/Promises work in other languages.

It requires a different mindset, different APIs, and different patterns. This needs to be more emphasized and explained.

That doesn't seem very idiomatic: it should be using futures::channel::mpsc rather than handling everything manually.

I would implement it like this:

pub struct WsIo
{
    ws: Rc< WebSocket >,

    pharos: Rc<RefCell< Pharos<WsEvent> >>,

    receiver: UnboundedReceiver< WsMessage >,

    _on_mesg: Closure< dyn FnMut( MessageEvent ) >,
    _on_close: Closure< dyn FnMut( Event ) >,
}

impl WsIo
{
    /// Create a new WsIo.
    //
    pub fn new( ws: Rc<WebSocket>, pharos : Rc<RefCell< Pharos<WsEvent> >> ) -> Self
    {
        let (sender, receiver) = futures::channel::mpsc::unbounded();


        let on_mesg = {
            let sender = sender.clone();

            Closure::wrap( Box::new( move |msg_evt: MessageEvent|
            {
                trace!( "WsStream: message received!" );
                sender.unbounded_send( WsMessage::from( msg_evt ) ).unwrap();
            }) as Box< dyn FnMut( MessageEvent ) > )
        };

        ws.set_onmessage  ( Some( on_mesg.as_ref().unchecked_ref() ) );


        let on_close = Closure::wrap( Box::new( move |msg_evt: MessageEvent|
        {
            trace!( "WsStream: closed!" );
            sender.close_channel();
        }) as Box< dyn FnMut( MessageEvent ) > );

        ws.set_onclose ( Some( on_close.as_ref().unchecked_ref() ) );


        Self
        {
            ws                  ,
            pharos              ,
            receiver            ,
            _on_mesg: on_mesg   ,
            _on_close: on_close ,
        }
    }
}

impl Drop for WsIo
{
    // We don't block here, just tell the browser to close the connection and move on.
    //
    fn drop( &mut self )
    {
        trace!( "Drop WsIo" );

        self.ws.set_onmessage( None );
        self.ws.set_onclose( None );
        // This can't fail
        //
        self.ws.close_with_code( 1000 ).expect( "WsIo::drop - close ws socket" );
    }
}

impl Unpin for WsIo {}

impl Stream for WsIo
{
    type Item = WsMessage;

    // Currently requires an unfortunate copy from Js memory to Wasm memory. Hopefully one
    // day we will be able to receive the MessageEvt directly in Wasm.
    //
    fn poll_next( mut self: Pin<&mut Self>, cx: &mut Context ) -> Poll<Option< Self::Item >>
    {
        trace!( "WsIo as Stream gets polled" );
        self.receiver.poll_next_unpin( cx )
    }
}
1 Like
14

Yeah, I didn't give many details. This example is an object that handles remote actors. So it's not just something that can implement Sink/Stream of messages. It deserializes incoming messages to different types and dispatches those to the correct actor.

It is an actor itself, and the user will have to move that into the mailbox responsible for running it. Thus it cannot be a future that resolves after the connection closes.

Concurrently we need to handle the incoming messages over the connection. This code does use a channel internally, but some task still has to listen to the channel.

The situation with ws_stream_wasm is that WsIo is a Sink/Stream of websocket messages + AsyncRead/AsyncWrite. The problem is that when you pass that to combinators or codecs, they will consume the object.

That means all the other API's from the websocket API are no longer available. This is solved by splitting it in 2 objects, with WsStream dealing with all the metadata and API's, and WsIo just being a simple Steam/Sink. So the on_close callback is taken by WsStream which exposes events in a rust way, eg. Stream of events. So unless I implement a callback type interface to notify WsIo on close, WsIo has to listen to this stream for the close event, which needs to run concurrently from the constructor returning the object...

15

I'm just thinking about something else, if you build one big future by await and join, you'll never be able to leverage a multithreaded task scheduler to spread the work over several cores.

16

External concurrency via spawning and internal concurrency via combinators are complementary techniques, choosing between them strongly depends on the usecase.

There is no guarantee that internal concurrency will be faster than spawning futures. For example join_all is a very naive combinator that results in O(n^2) polling, so if you wrap a lot of futures into it you will get a lot of overhead from polling them all. It is a niche type for use in cases where you are guaranteed to have a low number of homogeneous futures, but you don't know exactly what that number is to use one of the joinN functions, and can't use join! for some reason.

On the opposite side FuturesUnordered is basically an entire executor running inside your future. That means you're paying the waker system overhead twice for any future being polled inside it, keeping track of the sub-futures and knowing which to poll is not free. It's likely that a top-level executor is able to be more optimized for the specific target and have a lower overhead waker system, so spawning the sub-futures directly onto that when you don't need their results could be faster.

runtime also provides runtime::task::spawn and runtime::task::Spawner to allow runtime managed concurrency.

Another interesting point to this is that a library can rely on external concurrency by taking an impl Spawn, then a user of the library can convert that to internal concurrency within their futures by passing in an instance of FuturesUnordered<FutureObj> because of this Spawn implementation. Just because the library asks for a way to perform concurrency doesn't necessarily mean that that has to be handled by the runtime.

7 Likes
17

You can create something very similar to this by wrapping a spawner in another one that tracks all sub-futures it spawns. Without some form of AsyncDrop you would have to explicitly join that spawner to wait for all the sub-futures to complete, but maybe there would be a way to deal with it via some sort of closure based API.

Here's a non-trivial example using a wrapper nursery to scope some tasks while using a top-level spawner to actually run them: Rust Playground

1 Like
18

Thanks alot for the feedback. I kind of started to see if I could implement a nursery, but ended up with wrapping FuturesUnordered. I didn't know that AssertUnwindSafe implemented Future. That's pretty cool.

19

I still don't fully understand, but I suspect there's a way to fix it without spawning.

That's a good point, parallelism is at the Task level, not the Future level.

But I have a hard time imagining a single library Future that is so large that it would need to do that. Even with things like Tokio you spawn things at the top level of your application.

But in that hypothetical situation, yeah that would qualify as one of the "very rare" situations I mentioned.

20

To be clear, we're talking about spawning within a library. Obviously there are benefits to spawning for applications.

Yes, and in that case you should use FuturesUnordered, that has always been the recommendation.

That's possible, but it's also possible that FuturesUnordered is faster. It will depend on a lot of things.

In any case, it avoids the extra complexity of needing to pass in a Spawner, everything can be handled internally without needing to change the public API.

Yes, but we're talking about libraries, not applications. I have no problem with applications using spawn.

That's a good point. That doesn't change the fact that passing in impl Spawn adds in a lot of extra complexity.

And given how different Rust Futures are from other languages, many people will abuse spawning because that's what they're familiar with, rather than doing things in a better and more efficient Rust way.

So I still recommend that people try really hard to avoid spawning (in libraries) unless they're in one of the very rare situations where it's a good/necessary idea.