How to read file by chunks?

Hi guys! I have a small problem.
I need to read big file in chunks, and then calculate and SHA1 hash from every chunk.
I found Buffreader but it is very hard to understand how to.

I tried this:

let f = File::open("log.txt")?;
let reader = BufReader::with_capacity(10, f);

let mut m = sha1::Sha1::new();
m.update(reader);
println!("{}", m.digest().to_string());

but it wont work.. :confused:

It would be much easier for us to comment without having to recreate your results. So please also show the error you're getting, or the wrong behavior you're observing.

Error i am getting is: https://hastebin.com/ehehexaber.php

@fasae you're approaching this completely wrong.

The error you are getting is because you are trying to pass a BufReader into a method which takes a byte buffer (&[u8]). This means that you need to pass the bytes themselves into the m.update() method rather than the reader. Get used to understanding those types of compiler errors.

You can use reader.read_exact() to accomplish the task you are trying to accomplish, but you will need to pass in a buffer of the "chunk" size that you want to capture like this:

    use std::io::Read;
    use std::io::BufRead;
    pub fn read_n<T>(br: &mut T, num_bytes: usize) -> Result<Vec<u8>, ::std::io::Error>
    where T: BufRead
    {
        let mut buf = vec![0u8; num_bytes];
        let result = br.read_exact(&mut buf);
        match result
        {
            Ok(r) => return Ok(buf.to_vec()),
            Err(e) => return Err(e)
        }


    }

Most critical line to understand here is this: let mut buf = vec![0u8; num_bytes]; . The size of this buffer is what dictates how many bytes are read into each "chunk."
You can make your own version but for my uses I needed to return a Vec<u8> Result.

solution without BufRead, using Vec directly:

extern crate sha1;

use std::fs::File;
use std::io::Read;

fn main() -> Result<(), std::io::Error> {
    let mut f = File::open("log.txt")?;
    let mut data: Vec<u8> = Vec::new();
    f.read_to_end(&mut data);
    let mut m = sha1::Sha1::new();
    println!("{}", m.digest().to_string());
    Ok(())
}

please also note that sha1 has been broken, and there's a sha3 available:
https://crates.io/crates/sha3

1 Like