I am trying to write a simple personal web crawler. Admittedly, Rust might not be the ideal language for that, but I thought that it should definitely be doable.
Out of personal preference, I would like to do async I/O. However, I couldn’t find a crate that allows me to do asynchronous https requests without reinventing the wheel myself.
Reqwest (https://crates.io/crates/reqwest) and curl-rust (github. com/alexcrichton/curl-rust) seem to be only synchronous.
Both Hyper (hyper. rs/) and async-http-client (https://crates.io/crates/async-http-client), however, requires me to care about SSL Connectors and HTTP301 redirects and low-level stuff like that which I really don’t want to care about. I just want to get the bytes at a specific URL, without writing the upper part of the network stack myself. (Might be exaggerating here.) Basically, I am missing an async curl.
If anyone can suggest a suitable crate, I will be extremely grateful. Otherwise, it’s back to synchronous I/O for me.