How to stream async `reqwest::Response` to `GzDecoder`?

[winston0410]

I am trying to extract the downloaded tarbell from reqwest with flate2::bufread::GzDecoder. I have seen a very similar question, but that doesn't work as only reqwest::blocking::Response has implemented the Read trait.

How to stream reqwest response to a gzip decoder?

I am downloading a large gzip'd file with reqwest, decompressing it, and then parsing it in a manner similar to this: // client created with: reqwest::blocking::Client::builder() let gz_compressed_bytes = client.get(url).send()?.bytes()?; let mut gz_decompressed_bytes = vec![]; let mut gz = MultiGzDecoder::new(&gz_compressed_bytes[..]); gz.read_to_end(&mut gz_decompressed_bytes).expect("gz.read_to_end failed"); let remote_data: SomeRemoteData = serde_json::from_slice(&gz_decompressed_byte…

And for reqwest::Response, it hasn't implemented BufRead or Read. How can I pass the response body as byte into the GzDecoder?

[LegionMammal978]

It looks like flate2 removed async support in PR #292, recommending the now-popular async-compression crate as an alternative. With it, you have to do a bit of work to get a GzipDecoder:

[dependencies]
async-compression = { version = "0.3.12", features = ["futures-io", "gzip"] }
futures = "0.3.21"
reqwest = { version = "0.11.10", features = ["stream"] }
tokio = { version = "0.17.0", features = ["full"] }

use std::error;

#[tokio::main]
async fn main() -> Result<(), Box<dyn error::Error>> {
    use async_compression::futures::bufread::GzipDecoder;
    use futures::{
        io::{self, BufReader, ErrorKind},
        prelude::*,
    };
    let response = reqwest::get("http://localhost:8000/test.txt.gz").await?;
    let reader = response
        .bytes_stream()
        .map_err(|e| io::Error::new(ErrorKind::Other, e))
        .into_async_read();
    let mut decoder = GzipDecoder::new(BufReader::new(reader));
    let mut data = String::new();
    decoder.read_to_string(&mut data).await?;
    println!("{data:?}");
    Ok(())
}

[winston0410]

@LegionMammal978

async-compression

I don't mind switch crate and this is perfect. Thanks!