Tokio: how to `read_until_would_block`?

1

I have an AsyncRead. I want to process data from it in an interactive, streaming fashion, as it is produced.

It feels natural that the "chunk" of data I should process is whatever could be read without blocking. So I want an API like this:

pub async fn read_until_would_block(&mut self, buf: &mut Vec<u8>) -> io::Result<()>

I can not find the relevant API/pattern in the docs, does anyone know how to do it ^^ ?

2

Do you mean something like this?

1 Like
3

You can use something along these lines:

futures::future::poll_fn(|cx| { Poll::Ready(self.poll_read(cx)) }).await

This will call poll_read once and return immediately no matter what. It returns a Poll<Result<()>>.

It is also worth thinking about whether you want to use tokio::task::unconstrained.

3 Likes
4

Heh, I hoped for some existing utility! “Wait until some data is available, and then read all available data” sounds like a pretty primitive operation for an async world! But cooking up my own solution is also fine!

5

That kind of operation can be pretty dangerous - a sufficiently fast writer can OOM you. Reading just as much as you need and working on that avoids that by using the rest of the application logic as backpressure.

Things like the Codec API can help make that easier when trying to parse frames out of a stream: https://docs.rs/tokio-util/latest/tokio_util/codec/index.html

4 Likes
6

That's an excellent point, that's totally something I should handle, because that's for a terminal-shaped thingy...

This reminds me of a cute pattern in the Zig standard library, where all read_to_end-shaped APIs always takes an extra max_len argument as an upper bound on the amount of data you expect to read worst-case...