Simultaneously hashing and stream unzipping a reqwest response

Hi! Looking for some help. For fun, I'm migrating some Haskell code to Rust, and I don't know how to hand the bytes of an incoming stream (response) off to two consumers (zip::read::read_zipfile_from_stream and sha1::Sha1 simultaneously:

debug!("Attempting to unzip stream");
while let Ok(Some(z)) = read_zipfile_from_stream(&mut response) {
    debug!("Stream unzipping file {}", z.name());
    let mut out_file = File::create(z.name())?;
    let mut resp_buf = BufReader::with_capacity(buffer_size, z);

    loop {
        let buffer = resp_buf.fill_buf()?;
        out_file.write_all(buffer)?;
        if hash {
            hasher.update(buffer);
        }

        let length = buffer.len();
        resp_buf.consume(length);
        if length == 0 {
            break;
        }
    }
}

The problem is that the bytes I can see in z are not all of the bytes coming from response. I assume this is because zip::read::read_zipfile_from_stream consumes some bytes internally before handing them off to z. The result is that the hash I calculate via hasher is incorrect. The full code is here, for context (happy for any constructive criticism, BTW): https://gitlab.com/chjordan/giant-squid/-/blob/rust/src/asvo/mod.rs#L217

Do I need to use an mpsc or something from crossbeam (mpmc)? I honestly don't know how to proceed.

In Go I would use a TeeReader and there seems to be something similar in Rust. I can't find out if it's just unstable or deprecated/abandoned, but it doesn't seem to be in Nightly. There seems to be a crate offering similar functionality.

1 Like

Thanks for your reply! Sorry for the delay - busy.

I've been using that crate with some success. It seems to work as advertised, but not when I'm doing the stream unzipping. I'm not sure what I'm doing wrong (if anything), but I believe you have answered my question. Thanks again.