Hi, I recently began to learn Rust (after using mostly Python for the last couple of years). As part of my training, I translate some system scripts to Rust. Among these is a script which traverses a directory tree, creates hash digests for each file, and compares the digest to a baseline. I noticed that the Rust version of this script takes 3x longer to execute, and tracked the difference down to the hashlib / sha256 performance. The Python (test) script looks like this:
#!/usr/local/bin/python3
import hashlib
def get_hash(file_path):
myhash = hashlib.sha256()
with open(file_path,'rb') as file:
content = file.read()
myhash.update(content)
return myhash.hexdigest()
if __name__ == '__main__':
filepath = "/path/to/file"
get_hash(filepath)
The Rust code looks almost identical:
use sha256::try_digest;
use std::path::Path;
fn get_hash(filepath: &str) -> String {
let input = Path::new(filepath);
let hash_digest = try_digest(input).unwrap();
hash_digest
}
fn main() {
let filepath = "/path/to/file";
get_hash(filepath);
}
The test file at /path/to/file is 229 MB (to make the difference more pronounced), and a release build of the above Rust code takes 0.99 second to complete, while the Python code finished in 0.19 seconds. I am aware that Python relies on a C implementation for hashlib, but I still wonder why Rust takes more than 5x longer (in this isolated example) and 3x longer (in the real world scenario with many small files described above).
I am grateful for any hints. This was posted twice, as Akismet hid my first attempt after an initial reply pointing to caching. Unfortunately, while Rust does profit a bit from caching (0.83 after the initial 0.99), Python remains much faster (0.19) even if executed first.
What if you read the file once, then iterate hundreds of times hashing, and output the total time? This would reduce the measurement noise (from program loading and file reading), and make it easier to find precisely what is slower.
Edit: and even better, measure time inside the program instead of using an external tool.
Tried that, but this created a different problem. sha256::try_digest handles reading the file and calculating the digest, while sha256::digest requires a string (or byte string). So I tried
let buf = std::fs::read(filepath).unwrap();
let content = std::str::from_utf8(&buf).unwrap();
which causes an Utf8Error (the file in question is a gzipped database dump, which I usually would not convert to a string, but I found no way to derive a digest from the Vec directly). I still need to learn a lot about Rust.
I made a quick comparison by the way, reading the file into memory as a single blob, and measuring only the computation of the hash in-process. The results: Python 0.4s, Rust's sha256 crate: 0.6…0.7s.
I think this sha256 crate is simply a slower implementation. Try using another crate and see it it's faster. There's no intrinsic reason why the Rust version should be slower than the C underlying Python's hashlib.
use sha2::{Sha256, Digest};
use std::path::Path;
use std::fs::File;
use std::io;
fn get_hash(filepath: &Path) -> String {
let mut file = File::open(filepath).unwrap();
let mut hasher = Sha256::new();
io::copy(&mut file, &mut hasher).unwrap();
let digest = hasher.finalize();
format!("{:x}", digest)
}
But I tried both this and the "read the whole file" approach and the Pythonhashlib version is still faster. I think it's just a more optimized implementation (for the architecture I tried it on at least). RUSTFLAGS="-C target-cpu=native" and LTO and a single codeunit helped some, but not enough to catch up.
Thanks! I found a post on StackOverflow which provides both a solution for my misguided attempt at using sha256 with Vec and a working example for using io::copy in this context.
But as others pointed out in the meantime, the baseline (in terms of speed) is still the same.
Thank you! Now I am down to 0.17-0.19s, on par with Python's C-based lib (and learned quite a lot along the way).
FWIW, someone on the duplicate post (now unlisted) suggested using ring instead of sha2. The code is indeed slightly faster – 0.15-0.16s – (and a bit simpler) with ring:
let hasher = ring::digest::digest(&digest::SHA256, &bytes);
println!("{}", hex::encode(hasher));
Thanks again to everyone answering to my post (and not berating me for my ignorance).