I struggle with flate2 in combination with tokio

Hi folks,
I could really use some help here and feel like I must be missing something crucial.
I'm trying to create a dyn AsyncWrite + Send + Unpin from a GzEncoder for a given File.

First approach

Consider the following code:

use std::path::PathBuf;
use tokio::fs::create_dir_all;
use tokio::{
    fs::OpenOptions,
    io::{stderr, stdout, AsyncWrite, BufWriter},
};
use flate2::write::GzEncoder;
use flate2::Compression;
use std::io::Result;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum SupportedCompression {
    Uncompressed,
    Gzip,
}

async fn create_writer(
    path: &Option<PathBuf>,
    append: bool,
    compression: SupportedCompression,
    use_stdout: bool,
) -> Result<(Box<dyn AsyncWrite + Unpin + Send>, bool)> {
    if let Some(path) = path {
        let parent = path.parent();
        if let Some(parent) = parent {
            // make sure parent dir structure exists
            create_dir_all(parent).await?;
        }
        let mut file = OpenOptions::new()
            .write(true)
            .create(true)
            .append(append)
            .open(path)
            .await?;
            
        match compression {
            SupportedCompression::Uncompressed => {
                let writer = BufWriter::with_capacity(128 * 1024, file);
                Ok((Box::new(writer), true))
            }
            SupportedCompression::Gzip => {
                //let file = file.into_std().await;
                let writer = GzEncoder::new(file, Compression::default());
                Ok((Box::new(writer), true))
            }
        }
    } else {
        if use_stdout {
            Ok((Box::new(stdout()), false))
        } else {
            Ok((Box::new(stderr()), false))
        }
    }
}

Errors

This does not compile and causes the following errors:

error[E0277]: the trait bound `tokio::fs::File: std::io::Write` is not satisfied
  --> src/lib.rs:43:30
   |
43 |                 let writer = GzEncoder::new(file, Compression::default());
   |                              ^^^^^^^^^^^^^^ the trait `std::io::Write` is not implemented for `tokio::fs::File`
   |
   = note: required by `flate2::write::GzEncoder::<W>::new`

error[E0277]: the trait bound `tokio::fs::File: std::io::Write` is not satisfied
  --> src/lib.rs:43:30
   |
43 |                 let writer = GzEncoder::new(file, Compression::default());
   |                              ^^^^^^^^^ the trait `std::io::Write` is not implemented for `tokio::fs::File`
   | 
  ::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.20/src/gz/write.rs:39:25
   |
39 | pub struct GzEncoder<W: Write> {
   |                         ----- required by this bound in `flate2::write::GzEncoder`

error[E0277]: the trait bound `tokio::fs::File: std::io::Write` is not satisfied
  --> src/lib.rs:43:30
   |
43 |                 let writer = GzEncoder::new(file, Compression::default());
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::io::Write` is not implemented for `tokio::fs::File`
   | 
  ::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.20/src/gz/write.rs:39:25
   |
39 | pub struct GzEncoder<W: Write> {
   |                         ----- required by this bound in `flate2::write::GzEncoder`

error[E0277]: the trait bound `tokio::fs::File: std::io::Write` is not satisfied
  --> src/lib.rs:44:30
   |
44 |                 Ok((Box::new(writer), true))
   |                              ^^^^^^ the trait `std::io::Write` is not implemented for `tokio::fs::File`
   | 
  ::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.20/src/gz/write.rs:39:25
   |
39 | pub struct GzEncoder<W: Write> {
   |                         ----- required by this bound in `flate2::write::GzEncoder`

error[E0277]: the trait bound `flate2::write::GzEncoder<tokio::fs::File>: AsyncWrite` is not satisfied
  --> src/lib.rs:44:21
   |
44 |                 Ok((Box::new(writer), true))
   |                     ^^^^^^^^^^^^^^^^ the trait `AsyncWrite` is not implemented for `flate2::write::GzEncoder<tokio::fs::File>`
   |
   = note: required for the cast to the object type `dyn AsyncWrite + Send + Unpin`

error[E0277]: the trait bound `tokio::fs::File: std::io::Write` is not satisfied
  --> src/lib.rs:44:21
   |
44 |                 Ok((Box::new(writer), true))
   |                     ^^^^^^^^^^^^^^^^ the trait `std::io::Write` is not implemented for `tokio::fs::File`
   | 
  ::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.20/src/gz/write.rs:39:25
   |
39 | pub struct GzEncoder<W: Write> {
   |                         ----- required by this bound in `flate2::write::GzEncoder`

Alternative approach

If I uncomment the let file = file.into_std().await; line, it fails with this error instead:

Error

error[E0277]: the trait bound `flate2::write::GzEncoder<std::fs::File>: AsyncWrite` is not satisfied
  --> src/lib.rs:44:21
   |
44 |                 Ok((Box::new(writer), true))
   |                     ^^^^^^^^^^^^^^^^ the trait `AsyncWrite` is not implemented for `flate2::write::GzEncoder<std::fs::File>`
   |
   = note: required for the cast to the object type `dyn AsyncWrite + Send + Unpin`

Help! I'm really stuck!

The errors for both approaches make sense and I understand what they are saying...

I just fail to see how I can actually create a dyn AsyncWrite + Send + Unpin from a GzEncoder for a given file.

I'm trying to find a solution to this problem for quite a while now and suspect that I'm simply unaware of a very simple way out of this mess.

Any help would be appreciated!

You will not be able to use the flate2::write::GzEncoder type like this. It does not support asynchronous writers. Try the async-compression crate instead.

Unrelated to your issue, but Box<dyn AsyncWrite + Unpin + Send> is an anti-pattern. Use Pin<Box<dyn AsyncWrite + Send>> instead. It supports strictly more use-cases.

2 Likes

Actually, I just saw on the documentation for the flate2 crate that it claims to support async via a Tokio feature, but it isn't clear to me how.

Looking at the source code, the Tokio support is for Tokio 0.1.x, which is incredibly old. You will not be able to use the flate2 crate's async support in recent Tokio.

1 Like

Will try that, thanks!
I was under the impression that flate2 supports AsyncWrite with the "tokio" feature flag because of this paragraph in the docs.

Also thanks for pointing out the anti-pattern. These are my first steps in async Rust so it really helps. Clippy doesn't yet complain about it.

1 Like

Yes, that paragraph also confused me for a bit. Looking at the github repo, it seems like it's still accepting PRs. I'm sure they would accept a contribution that updates that paragraph to warn future readers about the version of Tokio.

Works like a charm! Thanks so much!

1 Like

I'll later open a bug there, I guess.
I'm not sure if they would rather implement support for current tokio than mentioning that the current async support is essentially obsolete right now.

Sure, but modifying the paragraph is a lot less work, so that's why I suggested that.

According to this comment and this issue they plan to deprecate the "tokio" feature altogether - which makes sense.

I should have checked the open issues earlier...

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.