Vector of async Tokio Commands

Hello Rustaceans,

I'm trying to run multiple commands at once using tokio::process::Command and since I don't know ahead of time how many command there will be, I want to put all the awaitables into a vector and then use join_all. But I'm issues with Rust complaining that the child doesn't live long enough. Here is what I have:

   let mut runnables: Vec<
        Pin<Box<dyn Future<Output = Result<ExitStatus, std::io::Error>> + Send>>,
    > = Vec::new();

    let mut command = Command::new("ls");
    command
        .stdout(Stdio::piped())
        .stderr(Stdio::piped());

    let mut child = match command.spawn() {
        Ok(child) => child,
        Err(err) => {
            eprintln!("Failed to spawn child process: {:?}", err);
            return Err(anyhow!(err));
        }
    };

    runnables.push(Box::pin(child.wait()));

    let results = join_all(runnables).await;

When I try to compile this, it tells me the borrowed value at child.wait() does not live long enough, pointing to the end of my function and saying: child` dropped here while still borrowed | borrow might be used here, when `runnables` is dropped and runs the `Drop` code for type `Vec

What am I getting wrong?

Is that really your code? I don't think that error should happen with it. I'm guessing that join_all is actually in a scope above the one in which the child is created. In that case, you will need the future to take ownership of child, like so:

runnables.push(Box::pin(async move { child.wait() }));

Edit: This was missing an await, the corrected version is:

runnables.push(Box::pin(async move { child.wait().await }));

Also, don't use join_all for this. join_all should almost never be used due to its quadratic time complexity; here, use a FuturesUnordered or FuturesOrdered, or spawn each child as a Tokio task, and then wait on each JoinHandle in sequence.

That is indeed my code. I took out the function header and end (which is just Ok(()) for now). I'm seeing that even with FuturesOrdered I'm needing to do the async move but I'm not sure I understand why that is. Can you explain this a little more? My FuturesOrdered is created in that same function and goes out of scope in that function, so I would have expected it to be fine since the child is also in and out of scope within that same function.

So I made some changes but I can't seem to find the right way to declare the type for the FuturesOrdered variable. Here is what I have:

In my lib.rs, I added:

#![feature(impl_trait_in_bindings)]

My new code:

    let mut runnables: FuturesOrdered<
        Pin<Box<impl futures::Future<Output = Result<ExitStatus, std::io::Error>>>>,
    > = FuturesOrdered::new();

    let mut command = Command::new("ls");
    command
        .stdout(Stdio::piped())
        .stderr(Stdio::piped());

    let mut child = match command.spawn() {
        Ok(child) => child,
        Err(err) => {
            slog::error!(logging::get(), "Failed to spawn child process: {:?}", err);
            return Err(anyhow!(err));
        }
    };

    runnables.push(Box::pin(async move { child.wait() }));

    loop {
        match runnables.next().await {
            Some(result) => {
                println!("    finished future [{:?}]", result);
            }
            None => {
                println!("Done!");
                break;
            }
        }
    }

The error I get:

     |
76   |         Pin<Box<impl futures::Future<Output = Result<ExitStatus, std::io::Error>>>>,
     |                 ----------------------------------------------------------------- the expected opaque type
...
98   |     runnables.push(Box::pin(async move { child.wait() }));
     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected opaque type, found a different opaque type
     |
     = note: expected opaque type `impl futures::Future` (opaque type at <src/lib/exec.rs:76:17>)
                found opaque type `impl futures::Future` (opaque type at <rust/library/core/src/future/mod.rs:61:43>)
     = note: distinct uses of `impl Trait` result in different opaque types

I think you need

async move { child.wait().await }

Note the await. Otherwise you’re trying to return a future that returns a future since the async block by itself is a future.

2 Likes

Additionally you don't need to use Nightly, just don't specify the type of runnables. And boxing the futures isn't necessary either, you can just store them inline since they're all the same type.

1 Like

I was thinking I'd throw some other async things in there as well (to gather output in a streaming fashion).

I''ve accepted your earlier post as the correct answer. For posterity, can you fix the child.wait() to be child.wait().await?

1 Like