Using async-std (was reqwest)

I am trying to use the simple example in reqwest but had run into a "dogs dinner" of problems with various mutually incompatible Rust editions and crate versions, etc.
Even using rustup beta did not help. Keep getting compiler complaints about async being unstable. Any simple instructions on how to actually make async work?

The async book talks about it as if it is supposed to work?

It used to work, but async got updated and reqwest wasn't.

1 Like

Thanks. Does that sort of thing happen often with Rust? Another question: so if I want to use anything non-blocking, specifically with slow http requests, I have to fall back to something else than Rust?

The latest released version of reqwest (0.9.something) has reqwest::async::Client, which uses futures 0.1.

The introduction of std::future is brand new, and the master branch of reqwest is being updated to use it. You can try it out with a nightly browser (await will be stable in Rust 1.39) and setting reqwest = { git = "https://github.com/seanmonstar/reqwest" } dependency in your Cargo.toml.

1 Like

It's mostly a problem right now because a very large language change is in progress right now.

The way I see it is that there's essentially two different worlds in Rust. The "bedrock" and "things that the Lang Team talk about on their streams".

If you stick to "bedrock" things are very unlikely to change. If you use crates that use things the Lang Team are still talking about, then it's likely that things are in some state of flux.

Regarding async: It very recently became stabilized. It's not in bedrock yet, but its place in the bedrock has been reserved. Those who are really invested in Rust have probably updated their crates for "final async". Others are probably waiting for Rust 1.39.

Personally I've put my async work on hold until 1.39. I expect that using things like reqwest after Rust 1.39 will be smooth sailing.

If you want to track 1.39: 1.39 Milestone · GitHub

Or for tracking async specifically: https://areweasyncyet.rs/

3 Likes

I recently found this post which I found very helpful on an experience report of Async/Await. I am not going to post it all here, but here is the start.

The Decision to Use Async/Await

Back before we made the decision to switch to using Async/Await and Futures 0.3 we were already using Futures 0.1 in the network stack and in many other layers of the system. As I’m sure most of you know asynchronous programming using Futures 0.1 is mostly done via using a set of combinators or by hand writing a state machine yourself. Most people don’t want to hand write state machines so most of our asynchronous code was written using combinators. One of the bigger challenges with using combinators is the inability to borrow as you normally would in non-futures rust code. This means that if you have a data structure that needs to be shared and used in the bodies of multiple combinators that you either have to pipe it through the input of each combinator and output of each produced future or you take the easy route and end up using lots of Arc<Mutex<T>> all over the place. A very overly simplified example would look something like this:

let shared = Arc::new(Mutex::new(HashMap::new()));
let shared_cloned = shared.clone();

let f = stream.for_each(move |item| {
    // Do something with `item` and produce another future
    
    ... shared.lock() ...
})
.and_then(|item| {
    // Do something with `item` and produce another future
    
    ... shared_cloned.lock() ...
})
.map_err(|err| {
    // All tasks must have an `Error` type of `()`.
});

tokio::spawn(f);
1 Like

No, the chaos with async is new, and hopefully temporary :slight_smile:

Rust editions work together seamlessly. As long as you have a not-totally-outdated compiler, all code — old and new — should just work, and you shouldn't even notice anything changed.

async/Futures incompatibility is different. Crates are currently be switching from the old futures 0.1 to the new async/await std Futures. The two different kinds of futures may be made to work together, but it's a bit of a pain, so it's better to avoid.

For now I suggest to stick to crates that use futures v0.1 (i.e. 95% of them). Avoid new/beta/experimental crates that switch over to std futures, until they're out of beta. During transition period it will be a bit of a mess.

2 Likes

I was lured into this by the talked about advantages of async/await, one of which was said to be "simplicity".

I accept what is being said about the flux.

Even so, may I make a general appeal to everyone who is, at any stage of development, purporting to provide examples, that they include complete and working Cargo.toml with all the correct dependencies and generally ensure that the example is actually self-contained and capable of running?

It has been "hell on wheels" trying to make any of them run. For example, nobody even mentions the indispensable executor, without which nothing will run, much less showing some working code with it.

Maybe I went into this with simplistic expectations: "I/O (synchronous or asynchronous) should be just a matter of some fairly standard boiler-plate. I will find some and build my code around it." I am sorry to say that in several days of research, I had failed to find any such workable boilerplate.

2 Likes

I agree that the current state of documentation is poor. My recommendation for choosing an executor is to just always use tokio::runtime::Runtime.

As for choosing the version of the library, you have to know whether you want legacy futures or new futures. Here are the version numbers for a few different libraries:

  • tokio: Legacy is 0.1.* and new is 0.2.0-alpha-*.
  • futures: Legacy is in the futures crate, new is in the futures-preview crate.
  • hyper: Legacy is 0.12.*, new is 0.13.0-alpha-*.
  • reqwest: Only legacy is available.

As for where to look for different features:

  • I/O: tokio-fs, tokio-net, tokio-io
  • Executor: tokio
  • Timers and sleeping: tokio-timer
  • Combining futures into one: futures
  • Various utilities for creating futures: futures
  • Various traits: futures
  • Asynchronous message passing: futures (note that both oneshot and ordinary channels are available)
  • HTTP and web: hyper or reqwest
4 Likes

I found a promising looking crate runtime. It is the only one out of all that lot that actually does anything for me. With rustup nightly.

According to this post by one of the authors, runtime is effectively abandoned. The recommendation for tokio, above, is a pretty safe one.

2 Likes

I get the impression that the best way to overcome incompatibility would be for the community to as a first step switch everything current to futures01. I've not see any such push (not looked hard.) Ideal world everything switches to async, but historic reality say this is likely multi year before the old gets buried. futures01 remains a strong stable library but keeping the futures crate as a 0.1 dependency will clash badly when 0.3 arrives.

I am a beginner who just wants to get some I/O working simply and efficiently over http. Rather than investing time into learning ways of doing it under complicated, inferior and soon to be obsolete "standard" crates, and then in a few months having to redo it all again, I want to go now straight to the best, which is clearly await/async.

The biggest problem I have encountered is that all learning examples and documentation refer only to randomly different stages of that existing obsolescence.

@Alice: referring me to Tokio is all well and good but none of their documentation or examples are await/async ready, not even in their alpha version repository.

@inejge: thank you for that update, it looks like the active and better documented async-std is what I need.

1 Like

@Rustafarian You can use my crate yukikaze which is already supporting std futures and is based on hyper too

As for tokio, the simple way to use it would be just:

[tokio::main]
async fn main() {
}

Which you can see even in simple example tokio - Rust

1 Like

Here's an example that downloads the contents of some web page using the new futures:

use tokio::runtime::Runtime;
use tokio::fs::File;
use tokio::prelude::*;

use hyper_tls::HttpsConnector;
use hyper::Client;
use hyper::Uri;

use std::error::Error;

fn main() {
    let runtime = Runtime::new().unwrap();
    runtime.spawn(async {
        match fetch_file().await {
            Ok(()) => { /* Success! */ },
            Err(err) => {
                println!("Error: {}", err);
            },
        }
    });
    runtime.shutdown_on_idle();
}

async fn fetch_file() -> Result<(), Box<dyn Error>> {
    let mut output = File::create("web.html").await?;

    // Using the hyper-tls crate allows us to access https pages.
    let https = HttpsConnector::new()?;
    let client = Client::builder().build::<_, hyper::Body>(https);

    // This gives us the headers of the response.
    let mut response = client.get(Uri::from_static("https://httpbin.org/ip")).await?;

    if !response.status().is_success() {
        println!("Failed to fetch url: {}.", response.status());
        return Ok(());
    }

    // Body is a stream of chunks of bytes.
    let body = response.body_mut();

    while let Some(chunk) = body.next().await {
        let bytes = chunk?.into_bytes();
        output.write_all(&bytes[..]).await?;
    }
    Ok(())
}

With Cargo.toml:

[package]
name = "dl_page"
version = "0.1.0"
authors = ["Alice Ryhl <alice@ryhl.io>"]
edition = "2018"

[dependencies]
tokio = "0.2.0-alpha.4"
hyper = "0.13.0-alpha.1"
hyper-tls = "0.4.0-alpha.1"
# This crate will be available as futures inside Rust.
futures-preview = "0.3.0-alpha.18"
5 Likes

The renaming in your Cargo.toml is unnecessary, the futures-preview crate distributes a library called futures (this is a relatively unused cargo feature where the library name doesn't have to match the crate name).

2 Likes

Thanks @anon15139276, this looks just like what I need! There is no denying that it is simpler than the "unofficial-official" offering and I have learned to value simplicity in this quest :grinning:

3 Likes

Sorry, I need some more help with this.

I had a synchronous output of a String to a file, which works fine using the write! macro from std::io::Write. (The data is really in my own defined struct converted to a String using Display but that is not important here).

Is there any equivalent write! macro in async-std?

I keep getting lifetime problems in the async blocks with all my variables, saying that they don't live long enough. But then how am I supposed to get that String value written to the file from the async block? Bear in mind that it is especially impossible to create a static lifetime value (as demanded) from a String. All the examples use only literal strings, which bypasses this problem.

Is there some trick to this that I am missing?

I'm puzzled as to how you are getting lifetime problems, can you post some example code with the issue?

Both the tokio and the async-std ecosystem provides AsyncWrite traits, which provides the functionality of writing data to a file. However, the standard traits only provide poll_write and friends, which are low-level and therefore not easily usable from async code.

I know that tokio provides a wrapper that solves this, so with that ecosystem you could create a String and use the write_all method to create a future that completes once all of the data has been written. If you don't already have a string, you can make one using format!. There's an example of using this in the code I posted earlier in this thread.

I'm not familiar with the async-std ecosystem, and I don't see any equivalent wrapper to the one from tokio, but I may just not be familiar enough to locate it.

1 Like