I'm designing an application in rust with producers that produce data, and consumers that consume it. Example of producers are a wrapper over network io and a gamepad producing events. Examples of consumers are also a wrapper over network io and a UI view. I'm considering two possible ways of doing this:
Producers use Stream abstraction, consumers use Sink abstraction. Stream combinators manipulate data. stream.forward(sink) moves data from the stream to the sink.
Producers expose an async read fn to read data. Consumers provide a regular write fn to accept data. In my application a handler fn reads items from the source one-by-one, manipulates them, and writes them to the sink.
Note that I do not require backpressure from sinks.
One inconvenience of approach (1) is there isn't anything provided in StreamExt for sending data from the same stream to multiple sinks. But this is trivial under (2) assuming items can be cloned.
I would be interested to hear reasons on why to prefer one approach over the other.
I'm also considering packaging these modules into webassembly and wonder if that would influence the answer. With approach (1) I am writing regular fns (not async) at the module level, and only deal with async at the top-level of the application. There is no async code burried in my producers or consumers. I'm not sure if this is of any benefit when interoperating with JS?
It sounds like you are effectively creating an actor system. @alice has an excellent blog post about how you might implement this without any extra frameworks or magic - it's just async Rust and channels. I'd recommend having a ready through and seeing if the ideas fit your application.
Having the main logic be synchronous can make things easier, but Rust, JavaScript, and wasm-bindgen all have good support for async code, so it shouldn't make much of a difference.
If, by "WebAssembly", you were planning to use this module outside of a browser (e.g. using a runtime like Wasmer or Wasmtime) then keeping your logic synchronous makes a massive difference
WebAssembly doesn't actually have a concept of async or futures. Instead, tools like wasm-bindgen will add glue code to both the host (the application running your WebAssembly) and guest (the Rust you compiled to a WebAssembly module) that makes everything work.
I know Wasmtime's wit-bindgen used to have support for async functions, but I just had another skim through the repo and docs, and it looks like it may have been removed.
Thanks for the detailed reply - especially the info on wasm runtimes.
I have watched Alice's talk on building actor's with tokio. It's a great resource, and it inspired me to take the actor model approach on a separate but related service that I am building. It was a good fit for this because I have components ('actors' I guess) that run as background services, and therefore need to run on their own task. Some examples include a component that services incoming requests from a cloud IOT platform, and a component that talks to external hardware over CANbus. These modules spawn their own task so they can keep doing things in the background - without the top level application passing flow control to them.
In contrast, the code I am writing now are modules that will go into a client application with a frontend. I currently don't have any real need for these modules to be running in the background on their own tasks. I'm considering whether it might be simpler if they only get flow control when the top level application polls for some data (provider) or pushes some data (sink).
From what you're saying this approach might have some benefits if I intend to run outside the browser. Currently we're only targeting the browser so maybe it doesn't make a big difference. But it's still tempting to try to keep other options open.
Channels are another option. This would change the flow-control in the application quite significantly. producers and consumers would need to spawn their own tasks for reading/writing to channels - rather than obtaining flow control from the main application when they are polled/written.