Tokio's AsyncReadExt and AsyncWriteExt require Self: Unpin. Why and what to do about it?

In my quest to understand asynchronous Rust better, I had a look at the asynchronous reader and writer traits that Tokio provides.

When I wanted to pass an asynchronous reader or writer to a generic function, I noticed that the methods I need are only implemented if the reader or writer is Unpin, see AsyncReadExt::read and AsyncWriteExt::write.

Why is that, and what do I need to do about it?

  • Declare my function as: async fn myfunc<W: AsyncWrite + Unpin>
  • Use tokio::pin!

What is the right thing to do?

The following code seems to work:

use tokio::io::AsyncWriteExt as _;

async fn myfunc<W>(writer: &mut W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,
    W: std::marker::Unpin,
{
    writer.write(b"ABC\n").await?;
    Ok(())
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    myfunc(&mut tokio::io::stdout()).await?;
    Ok(())
}

Naïvely trying to use tokio::pin! on the writer (reference) does NOT work:

async fn myfunc<W>(writer: &mut W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,
{
    tokio::pin!(writer);
    writer.write(b"ABC\n").await?;
    Ok(())
}

I get: error[E0277]: `W` cannot be unpinned
I assume that is because writer is a mutable reference, and not the writer itself (that needs to be Unpin).

If I declare the function to consume the writer, the code compiles and works fine:

use tokio::io::AsyncWriteExt as _;

async fn myfunc<W>(writer: W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,
{
    tokio::pin!(writer);
    writer.write(b"ABC\n").await?;
    Ok(())
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    myfunc(&mut tokio::io::stdout()).await?;
    Ok(())
}

Note that I can still pass a &mut reference to myfunc; I guess that's because &mut T, where T: ?Sized + AsyncWrite + Unpin, automatically implements AsyncWrite as well.

What is the correct way to write myfunc here? Expect the writer to be Unpin or consume the writer and pin it myself? I did not find any documentation yet on these issues. I also don't understand the reasons why AsyncWriteExt is only implemented for async writers that are Unpin.

Can someone enlighten me or tell me what's the idiomatic way of passing an async reader or writer to a function that needs to use the AsyncReadExt or AsyncWriteExt methods?

Is this issue documented somewhere?

Another option is to take Pin<&mut W>.

use std::pin::Pin;
use tokio::io::AsyncWriteExt;
async fn myfunc<W>(mut writer: Pin<&mut W>) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,
{
    writer.write(b"ABC\n").await?;
    Ok(())
}

Ultimately it doesn’t matter all that much though anyways, since

async fn myfunc<W>(writer: W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,

can still be called with &mut W where W: Unpin because of this impl

impl<T: ?Sized + AsyncRead + Unpin> AsyncRead for &mut T

And conversely, this function

async fn myfunc<W>(writer: &mut W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,
    W: std::marker::Unpin,

can still be called with an owned W: !Unpin because the caller can just use tokio::pin!, then pass a &mut Pin<&mut W>, because if this impl

impl<P> AsyncRead for Pin<P>
where
    P: DerefMut + Unpin,
    P::Target: AsyncRead, 

(with P == &mut W, note that &mut W: Unpin even when W: !Unpin)


Note that the signature of the function I presented in my previous answer:

async fn myfunc<W>(mut writer: Pin<&mut W>) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,

would be a special case of

async fn myfunc<W>(writer: W) -> tokio::io::Result<()>
where
    W: tokio::io::AsyncWrite,

because that one can accept a Pin<&mut W> qualifies as an “owned” writer as-well. (Again, because of the same AsyncRead for Pin<P> implementation mentioned above.)

1 Like

The reason behind this choice is that there are basically two choices:

  1. Make the AsyncWriteExt methods take &mut W and require Unpin.
  2. Make the AsyncWriteExt methods take Pin<&mut W> and don't require Unpin.

With the first choice you can use the methods normally for types that are Unpin, and for non-Unpin types, you must tokio::pin! it first. With the second choice, you must always tokio::pin! it. Hence the first choice is more convenient.

You can't tokio::pin! given only a &mut W because you have no way of preventing the caller from moving the W after the call. Pinning requires ownership.

1 Like

I don’t know what’s most idiomatic. My best guess is that AsyncRead or AsyncWrite implementors that aren’t Unpin are somewhat uncommon, and &mut T is a bit easier to work with than Pin<&mut T> (you’ll never need Pin::new-calls if you had T: Unpin types anyways, you don’t need to call as_mut for reborrowing, etc), so to make the common case more easy, these utility methods use &mut W (with W: Unpin) rather than Pin<&mut W> for convenience. And as demonstrated, passing Pin<&mut W> is still possible via &mut Pin<&mut W>. So maybe, if tokio does it for convenience in methods of AsyncReadExt, then you can take the same approach of just adding some W: Unpin bounds as-well.

Yes, if you need to take a reference instead of ownership, I'd go for &mut W with Unpin as well.

1 Like

For Unpin types, Pin::new works as-well. Really, it’s not too hard to transform either variant into the other

use std::pin::Pin;
use tokio::io::AsyncRead;

pub fn foo1<W: AsyncRead + Unpin>(x: &mut W) {
    // can call bar1:
    bar1(Pin::new(x))
}
pub fn bar1<W: AsyncRead>(_x: Pin<&mut W>) {}


pub fn bar2<W: AsyncRead>(x: Pin<&mut W>) {
    // can call foo2:
    foo2(&mut {x})
}
pub fn foo2<W: AsyncRead + Unpin>(_x: &mut W) {}

I can only see convenience for the common case (i.e. avoid need for Pin::new) as a reason, as well as – possibly – considerations of avoiding multiple steps of indirection. Though, indirection is only really added if you keep switching between the two styles, so I guess consistently using &mut W (with W: Unpin) is a reasonable approach.

Thanks. Now I see how these approaches can be transformed into one another. I still find it highly irritating though and feel like it requires a lot of extra impl's on the library (i.e. Tokio's) side.

I wonder if I would end up with a lot of these extra impl's when I create my own asynchronous functions that require Unpin. Maybe it will all feel less confusing once I get most accustomed to pinning and unpinning, but it certainly makes things very difficult for a beginner.

Is there any place in Tokio's documentation where these issues could be covered? Maybe in module tokio::io or in the I/O section of the Tokio tutorial?

I have thought a bit about writing something about how to use the IO traits with generics as a new chapter under the topics heading, but I don't think I'll get around to it any time soon.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.