How can I mock tokio::io::AsyncReadExt and AsyncWriteExt traits?

My goal is to have unit tests inject a factory trait (called Networker) implementation into my system under test that returns an opaque implementation of AsyncReadExt + AsyncWriteExt + Unpin from its Networker::connect() function such that I can have the same functions I need to call in tokio::net::TcpStream in the system under test behave arbitrarily to suit the test conditions. I'm hoping to use mockall to mock up the AsyncReadExt and AsyncWriteExt traits, which also requires mocking AsyncRead and AsyncWrite. I've gotten somewhat close with the following:

/**
 * Mockable trait that combines the same read/write traits we need our Networker::connect() DI
 * function to return in the system under test. This allows us to install test specific behavior via expectations
 * for the opaque read/write impl returned in the system under test and thereby explore TCP read/write edge cases.
 */
trait AsyncReadWriteStream: AsyncReadExt + AsyncWriteExt + Unpin {}
mock! {
    AsyncReadWriteStream {}
    impl AsyncRead for AsyncReadWriteStream {
        fn poll_read(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'_>,
            buf: &mut ReadBuf<'_>,
        ) -> Poll<Result<(), std::io::Error>>;
    }
    impl AsyncReadExt for AsyncReadWriteStream {
        fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize>;
    }
    impl AsyncWrite for AsyncReadWriteStream {
        fn poll_write(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'_>,
            buf: &[u8],
        ) -> Poll<Result<usize, std::io::Error>>;
        fn poll_flush(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'_>,
        ) -> Poll<Result<(), std::io::Error>>;
        fn poll_shutdown(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'_>,
        ) -> Poll<Result<(), std::io::Error>>;
    }
    impl AsyncWriteExt for AsyncReadWriteStream {
        fn write_all(&mut self, buf: &[u8]) -> std::io::Result<()>;
    }
}

This complains about wanting named lifetime params at the '_ points, so I added in some:

trait AsyncReadWriteStream: AsyncReadExt + AsyncWriteExt + Unpin {}
mock! {
    AsyncReadWriteStream {}
    impl AsyncRead for AsyncReadWriteStream {
        fn poll_read<'a>(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'a>,
            buf: &mut ReadBuf<'a>,
        ) -> Poll<Result<(), std::io::Error>>;
    }
    impl AsyncReadExt for AsyncReadWriteStream {
        fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize>;
    }
    impl AsyncWrite for AsyncReadWriteStream {
        fn poll_write<'a>(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'a>,
            buf: &[u8],
        ) -> Poll<Result<usize, std::io::Error>>;
        fn poll_flush<'a>(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'a>,
        ) -> Poll<Result<(), std::io::Error>>;
        fn poll_shutdown<'a>(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'a>,
        ) -> Poll<Result<(), std::io::Error>>;
    }
    impl AsyncWriteExt for AsyncReadWriteStream {
        fn write_all(&mut self, buf: &[u8]) -> std::io::Result<()>;
    }
}

But that gets me to a confusing "conflicting implementations" error:

error[E0119]: conflicting implementations of trait `tokio::io::AsyncWriteExt` for type `MockAsyncReadWriteStream`
    --> ...
     |
1627 | /     mock! {
1628 | |         AsyncReadWriteStream {}
1629 | |         impl AsyncRead for AsyncReadWriteStream {
1630 | |             fn poll_read<'a>(
...    |
1654 | |         impl AsyncWriteExt for AsyncReadWriteStream {
     | |___________________________________________________^
     |
     = note: conflicting implementation in crate `tokio`:
             - impl<W> tokio::io::AsyncWriteExt for W
               where W: tokio::io::AsyncWrite, W: ?Sized;
     = note: this error originates in the macro `mock` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0119]: conflicting implementations of trait `tokio::io::AsyncReadExt` for type `MockAsyncReadWriteStream`
    --> ...
     |
1627 | /     mock! {
1628 | |         AsyncReadWriteStream {}
1629 | |         impl AsyncRead for AsyncReadWriteStream {
1630 | |             fn poll_read<'a>(
...    |
1636 | |         impl AsyncReadExt for AsyncReadWriteStream {
     | |__________________________________________________^
     |
     = note: conflicting implementation in crate `tokio`:
             - impl<R> tokio::io::AsyncReadExt for R
               where R: tokio::io::AsyncRead, R: ?Sized;
     = note: this error originates in the macro `mock` (in Nightly builds, run with -Z macro-backtrace for more info)

error: `impl` item signature doesn't match `trait` item signature
    --> ...
     |
1627 | /     mock! {
1628 | |         AsyncReadWriteStream {}
1629 | |         impl AsyncRead for AsyncReadWriteStream {
1630 | |             fn poll_read<'a>(
...    |
1657 | |     }
     | |_____^ found `fn(Pin<&'1 mut MockAsyncReadWriteStream>, &'2 mut std::task::Context<'3>, &mut tokio::io::ReadBuf<'3>) -> std::task::Poll<std::result::Result<(), std::io::Error>>`
     |
    ::: C:\Users\jcreswell\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\tokio-1.44.2\src\io\async_read.rs:54:5
     |
54   | /     fn poll_read(
55   | |         self: Pin<&mut Self>,
56   | |         cx: &mut Context<'_>,
57   | |         buf: &mut ReadBuf<'_>,
58   | |     ) -> Poll<io::Result<()>>;
     | |______________________________- expected `fn(Pin<&'1 mut MockAsyncReadWriteStream>, &'2 mut std::task::Context<'3>, &mut tokio::io::ReadBuf<'_>) -> std::task::Poll<std::result::Result<(), std::io::Error>>`
     |
     = note: expected signature `fn(Pin<&'1 mut MockAsyncReadWriteStream>, &'2 mut std::task::Context<'3>, &mut tokio::io::ReadBuf<'_>) -> std::task::Poll<std::result::Result<(), std::io::Error>>`
                found signature `fn(Pin<&'1 mut MockAsyncReadWriteStream>, &'2 mut std::task::Context<'3>, &mut tokio::io::ReadBuf<'3>) -> std::task::Poll<std::result::Result<(), std::io::Error>>`
     = help: the lifetime requirements from the `impl` do not correspond to the requirements in the `trait`
     = help: verify the lifetime relationships in the `trait` and `impl` between the `self` argument, the other inputs and its output
     = note: this error originates in the macro `mock` (in Nightly builds, run with -Z macro-backtrace for more info)

and changing just ReadBuf param back to ReadBuf<'_> just reintroduces the "expected named lifetime parameter" error. I also tried using the exact function signatures for AsyncReadExt and AsyncWriteExt e.g. fn write_all<'a>(&'a mut self, src: &'a [u8]) -> WriteAll<'a, Self> but the return struct WriteAll is part of the private io::util module in tokio.

How should I set up mock!{} such that my mock implementation of AsyncReadExt, AsyncWriteExt, and their dependencies satisfy all the constraints of the tokio implementations?

I don't know the answer to your main question off-hand, but the last error is[1] due to this change:

     impl AsyncRead for AsyncReadWriteStream {
-        fn poll_read(
+        fn poll_read<'a>(
             self: Pin<&mut Self>,
-            cx: &mut std::task::Context<'_>,
-            buf: &mut ReadBuf<'_>,
+            cx: &mut std::task::Context<'a>,
+            buf: &mut ReadBuf<'a>,
         ) -> Poll<Result<(), std::io::Error>>;
     }

For that one, try preserving the original semantics by using distinct lifetimes.

    impl AsyncRead for AsyncReadWriteStream {
        fn poll_read<'ctx, 'buf>(
            self: Pin<&mut Self>,
            cx: &mut std::task::Context<'ctx>,
            buf: &mut ReadBuf<'buf>,
        ) -> Poll<Result<(), std::io::Error>>;
    }

  1. presumably, since we are talking about a macro here â†Šī¸Ž

1 Like

Looking at your main error now...

I think you just need to not implement AsyncWriteExt (and AsyncReadExt). The blanket implementation will provide the functionality for you once AsyncWrite (AsyncRead) is implemeted.

1 Like

What does blanket implementation mean, exactly? I looked at the AsyncWriteExt source and couldn't find any direct calls to poll_write(), but maybe it's buried in abstraction layers somewhere? If I can mock AsyncRead and AsyncWrite and then in the system under test call the *Ext convenience functions which will eventually call in to my mocked AsyncRead/AsyncWrite functions, I think that could do the trick? Provided the *Ext fellas don't need their AsyncRead/AsyncWrite implementation to be any more 'real' than having the expected function signature.

Are you aware of tokio_test::io (made by the tokio team), which already provides a mock AsyncRead & AsyncWrite type?

A blanket implementation of a trait Foo is an impl<T> Foo for T where T: SomeOtherTrait. In the case of AsyncReadExt, it's implemented for all types that implement AsyncRead + ?Sized, so just implementing AsyncRead on your type is enough to let you use the AsyncReadExt methods (though the Ext trait still has to be imported), and likewise for AsyncWrite.

2 Likes

In practical terms in means that upstream has provided an implementation that covers downstream types. In this particular case, any implementor of AsyncWrite uses the tokio implementation for AsyncWriteExt:

// This is in the `tokio` crate
impl<W> tokio::io::AsyncWriteExt for W // <-- It is a "blanket implementation"
where                                  // because it applies to any type --
    W: tokio::io::AsyncWrite,          // because this is an "uncovered" `W`
    W: ?Sized,                         // and not something like `Vec<W>`
{ ... }

At most one implementation is allowed per type, so if you try to provide your own implementation in addition to an AsyncWrite implementation, it must fail.

On top of that, AsyncWrite is a supertrait of AsyncWriteExt -- if you want AsyncWriteExt you must implement AsyncWrite -- and the end result is that the blanket implementation is the only implementation of AsyncWriteExt that allowed to exist.

(If we ever get specialization, we may get a way to opt into allowing overriding implementations, but we don't have that currently.)

In addition to whatever they do in the method bodies, the AsyncWriteExt methods will dispatch to the AsyncWrite implemention of the type -- that's all they can do. It's not about expected function signatures though, it's about having an actual implementation of the trait. There's no duck-typing here, and more generally, Rust tries to avoid "post-monomorphization errors".

In practical terms here that means that if your type implements AsyncWrite, it will then definitely implement AsyncWriteExt. And you can know that just from seeing that the blanket implementation exists. The implementation isn't allowed to make other hidden assumption that would only fail when you threw the wrong type at it. It's only allowed to use the properties of the type that were declared in the bounds -- here, that the type implements AsyncWrite.

1 Like

I took a quick look at tokio_test earlier, but it seems they went a totally different direction with their mock AsyncWrite/AsyncRead -- they let you call read and write functions in arbitrary sequences, but they don't let you install arbitrary read/write behavior as far as I could see (and understand). I'm specifically trying to contrive a scenario where a mocked call to read() returns very specific binary data and a mocked call to write() uses tokio::time::sleep() to stall and trip a monitoring tokio::time::timeout() function. So I think tokio_test is a little too rigid to meet my requirements? It seems designed to test tokio itself, as opposed to systems making use of tokio APIs.

In case I wanted to avoid AsyncWrite/AsyncRead conveniences entirely, how would I directly call AsyncRead and AsyncWrite poll_* functions? I'm not sure where the std::task::Context comes from; I know it provides the Waker for a task and somehow comes into being from the dark magics of executors, but I haven't found an example showing where the Context comes from in the first place (the usages within tokio mostly seem to be within functions that already accept a Context param from somewhere, so they can just forward it).

That sounds like mocking the executor territory. I'm afraid I'm the wrong person to ask about that.

1 Like

The only reasonable option I know of is to implement a Future and run it in tokio (eg with .await or block_on()), you will get the runtime's Context. A really easy way to do this is poll_fn

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.