AsyncWrite into Vec

I made a trait to allow binary export and import of some data types (Note: maybe I should use Serde, but it seemed to be a bit overkill for my scenario). Converting my project to async, using Tokio, I ended up with the following definition:

use async_trait::async_trait;
use std::marker::{Send, Unpin};
use tokio::io::{self, AsyncRead, AsyncWrite};

#[async_trait]
pub trait Binary {

    async fn dump<W>(&self, writer: &mut W) -> io::Result<()>
    where
        W: AsyncWrite + Unpin + Send;

    async fn restore<R>(&self, reader: &mut R) -> io::Result<Self>
    where
        Self: Sized,
        R: AsyncRead + Unpin + Send;

}

I also would like to provide a method where I can create a binary representation as a Vec<u8>, so I added the following method with a default implementation:

    fn dump_to_vec(&self) -> Vec<u8> {
        let mut result = Vec::new();
        let rt = tokio::runtime::Builder::new_current_thread()
            .build()
            .unwrap();
        rt.block_on(self.dump(&mut result)).unwrap();
        result
    }

Here comes my questions:

  1. Is it correct to not enable I/O using tokio::runtime::Builder::enable_io here?

  2. Is this basically the same as if I would use block_on of the futures crate? What are the differences between using a Tokio runtime and using block_on of the futures crate? In particular: Would there be some overhead if I create a Tokio Runtime?

Same code using futures crate:

    fn dump_to_vec(&self) -> Vec<u8> {
        let mut result = Vec::new();
        futures::executor::block_on(self.dump(&mut result)).unwrap();
        result
    }

Using the futures crate apparently makes the code shorter, but would introduce an additional dependency (unless there is a reason why I need futures anyway?).

The enable_io call enables the runtime's io driver. Using anything in tokio::net requires the IO driver and will fail if you try to use it on a runtime without the io driver.

Also, please be aware that it would be invalid to ever call your dump_to_vec method from the context of an async function because it is blocking.

It was a bit confusing that enable_io isn't required for tokio::io (both are "io"), but tokio::net. Its documentation isn't so specific:

Enables the I/O driver.

Doing this enables using net, process, signal, and some I/O types on the runtime.

Not sure what "some I/O types" are, but looks like AsyncRead(Ext) and AsyncWrite(Ext) are fine.

If my Binary::dump method above is only blocking because of I/O operations on the writer, then it should be effectively be turned into a non-blocking method when I pass &mut Vec<u8>, right? Thus I would assume I can call dump_to_vec from an async function (under the premise that the only blocking operations in dump are only on the writer), or am I wrong here?

Thank you for your above link to your blog (Async: What is blocking?) and the advice when to use a dedicated thread. That's very good to know.

So I assume I do not need enable_io if I use a Tokio runtime, but I'm still not sure whether there is any difference between

  • tokio::runtime::Builder::new_current_thread().build().unwrap().block_on

and

  • futures::executor::block_on.

I guess it's basically(?) the same? Or is there some extra magic in Tokio's runtime?

It is the IO types in tokio::net as well as tokio::io::unix::AsyncFd. If you use one of the IO traits you mention with one of these IO types, you will get a panic unless the IO driver is enabled. It's true that some of the features in tokio::process and tokio::signal also require the IO driver.

This is true, but it's best to entirely avoid going through block_on then. Tokio's block_on method will panic if you call it in the async context to prevent blocking the thread.

The IO driver isn't the only Tokio specific feature. There are various others as well, and some of them are always enabled. One example is tokio::spawn and spawn_blocking, which would only work inside a Tokio runtime.

:scream:
That's good to know! (And also explained in the documentation, I just missed that.)

I assume futures::executor::block_on, in contrast, would work in my case, also if I call it from an async method.

Hm, but I don't know what I could/should do instead to implement dump_to_vec without duplicating the logic of dump. When I know that the async function doesn't block when called with certain parameters, I don't see why my approach is a bad idea (other than that Tokio would panic here to keep me from doing something supposedly bad). So what I tried to do is just "bad style"?

Maybe I don't need the dump_to_vec method anyway (I will see when I have progressed further) and should not implement it at all.

But isn't it a common task to asynchronously write something that sometimes goes to real I/O interfaces and sometimes is just stored in memory for further processing? In the latter case, there is no need for a corresponding function to be asynchronous (and to make all callers of this function asynchronous as well). That is why I had the idea to provide a non-async variant of dump, where the destination is a Vec<u8>.

Perhaps in a real-world project, you end up with everything async anyway? (This does remind me of all Lua functions being asynchronous, by the way). I have too little experience with asynchronous programming to understand the usual "style" here.

Using block_on for this sort of purpose would indeed be bad style, but if you really want to do it, I would recommend this:

futures::executor::block_on(tokio::task::unconstrained(async { ... }));

However there would be better ways of doing this. For example, you might write your logic in a non-async function which will process the data you give it, then return. The async code can then alternate between waiting for more data and calling that non-async function. Your dump_to_vec method can then use that non-async function directly instead of going through block_on.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.