Unzip reqwest body as a stream

Hi,

I'm using reqwest to download files over the internet. These files may be very large and I want to stop reading the body if the size exceeds some threshold. In addition, they may be zipped with various http compression protocols (gzip, brotli, zlib).

Using the [reqwest::Response - Rust](https://bytes_stream method) in reqwest, it seems possible to read body content as a stream.

On the other hand, async_compression crate provides decoders which take streams as input.

However, I'm not able to connect both parts (even to make something that compiles).

Pseudo-code :

use tokio_util::io::StreamReader;
use async_compression::tokio::bufread::{BrotliDecoder, GzipDecoder, ZlibDecoder};
use tokio::io::{AsyncReadExt, Result};

async fn download(uri: reqwest::Url) {
        let res = self.client.get(uri.as_str()).send().await?;
        let stream = res.bytes_stream();
        let mut stream_reader = StreamReader::new(stream);
        let mut decoder = GzipDecoder::new(stream_reader);
       // here iterate on decoder, write bytes in some buffer until threshold is reached, catch errors and return
    }

However I can't get further than this kind of errors :

error[E0277]: the trait bound std::io::Error: From<reqwest::Error> is not satisfied
--> ***
|
252 | let mut stream_reader = StreamReader::new(stream);
| ^^^^^^^^^^^^^^^^^ the trait From<reqwest::Error> is not implemented for std::io::Error
|
= help: the following implementations were found:
<std::io::Error as From>
<std::io::Error as From>
<std::io::Error as From<IntoInnerError>>
<std::io::Error as From>
and 4 others
= note: required because of the requirements on the impl of Into<std::io::Error> for reqwest::Error
= note: required by StreamReader::<S, B>::new

error[E0277]: the trait bound std::io::Error: From<reqwest::Error> is not satisfied
--> ***
|
253 | let mut decoder = GzipDecoder::new(stream_reader);
| ^^^^^^^^^^^^^^^^ the trait From<reqwest::Error> is not implemented for std::io::Error
|
= help: the following implementations were found:
<std::io::Error as From>
<std::io::Error as From>
<std::io::Error as From<IntoInnerError>>
<std::io::Error as From>
and 4 others
= note: required because of the requirements on the impl of Into<std::io::Error> for reqwest::Error
= note: required because of the requirements on the impl of tokio::io::AsyncBufRead for StreamReader<impl Stream, bytes::bytes::Bytes>
= note: required by async_compression::tokio::bufread::GzipDecoder::<R>::new

This is because you must convert the error type manually.

use futures::stream::TryStreamExt; // for map_err

fn convert_error(err: reqwest::Error) -> std::io::Error {
    todo!()
}

let stream = res.bytes_stream().map_err(convert_error);
let mut stream_reader = StreamReader::new(stream);
let mut decoder = GzipDecoder::new(stream_reader);
1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.