I am using async/await everywhere in my code base. And wondering if there is a way to make external_crate::Foo<Write> support async/await with async Write?
In my case, external_crate::Foo<Write> is a struct that implements some compression algorithm, it wraps a Write trait object (e.g., File).
I can call foo.write_something() in my async function. However, since Writer writes to a file in my use case, it would be nice to use epoll based async file writer instead of sync file writer which make a blocking write system call that blocks the Tokio worker thread (does the performance difference actually matter?).
Is there a magic crate that provide an object that implement Write and underlying does async Write and at the same time works well with Tokio?
Not sure if I explained this clearly, would be happy to further clarify my question.
At least the Linux doesn't support true non-blocking fs operations. I'm not sure about windows, but both tokio and async-std implements fs operations using threadpool. So in your use case it should be near-optimal to use tokio::task::spawn_blocking().
I doubt it's possible to do that without adding a bunch of runtime overhead. The Write trait (which is required by the external crate) expects that when the write() method returns it's copied the data to the destination, whereas the AsyncWrite trait is asynchronous and needs to be polled to completion.
I guess you could hack something together using tokio::runtime::Runtime::block_ok() in Write::write() which will start the write on the underlying AsyncWrite and tell the tokio executor to keep running until the write operation completes. That would allow the runtime to execute other tasks while the write operation is running at least.
Sorry there's no magic to make sync code async. You might be able to create a custom Write impl that intercepts the data into a memory buffer, and then write that buffer using ordinary async io.
That said, as some of the others commented, file IO currently cannot be done in an async manner in async await, so if you're writing to files anyway, I'd just put the operation inside an spawn_blocking, and have Tokio schedule the blocking operation on one of its threads dedicated to blocking operations.