Easy async http crate

help
1

I am trying to write a simple personal web crawler. Admittedly, Rust might not be the ideal language for that, but I thought that it should definitely be doable.

Out of personal preference, I would like to do async I/O. However, I couldn't find a crate that allows me to do asynchronous https requests without reinventing the wheel myself.

Reqwest (https://crates.io/crates/reqwest) and curl-rust (github. com/alexcrichton/curl-rust) seem to be only synchronous.

Both Hyper (hyper. rs/) and async-http-client (https://crates.io/crates/async-http-client), however, requires me to care about SSL Connectors and HTTP301 redirects and low-level stuff like that which I really don't want to care about. I just want to get the bytes at a specific URL, without writing the upper part of the network stack myself. (Might be exaggerating here.) Basically, I am missing an async curl.

If anyone can suggest a suitable crate, I will be extremely grateful. Otherwise, it's back to synchronous I/O for me.

1 Like
2

Reqwest has async.

3 Likes
3

...but as the module name suggests, it is currently unstable and the interface may change in the future.

4

Since the underlying technology for async in Rust (tokio/futures) is still unstable, I don't think you'll be able to find any async HTTP client implementation that is committing to a stable API.

2 Likes
5

Wonderful! Thank you so much, this seems to be exactly what I was looking for.

6

curl-rust is both synchronous and asynchronous. The easy interface by itself is synchronous but using the multi interface it is asynchronous.

One part that is often overlooked however is DNS lookups are generally synchronous even on the multi interface and some other async clients.

I developed mio_httpc that is entirely async (DNS lookups as well): https://github.com/SergejJurecko/mio_httpc

However it may not be as high level as you want it to be.

1 Like