Ownership and async move inside a loop?

Hey guys, I'm really new to Rust and have banged my head for a few hours at this problem.

I'm using Hyper and Tokio, and am trying to write a simple router, but I cannot seem to wrap my head around the ownership going on. I couldn't find any relevant help pages, but that is partially because I don't even know what to search.

I've written a new stripped down version of what I'm trying to do (38 lines):

Playpen: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1b8761158e858924fc9909da7ffa44da

Code:

use hyper::server::conn::Http;
use hyper::service::service_fn;
use std::sync::Arc;
use tokio::net::TcpListener;
use tokio::stream::StreamExt;
use hyper::{Body, Response, StatusCode};

/** A webserver that responds with 200 if its path contains a string. */
#[tokio::main]
async fn main() {
    let strings_to_match = Arc::new(vec!["test", "hello", "yes"]);
    let http = Arc::new(Http::new());

    let mut listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    while let Some(stream) = listener.incoming().next().await {
        let conn = http.serve_connection(
            stream.unwrap(),
            service_fn(|req| async move {
                for string in *strings_to_match {
                    if req.uri().path().to_string().contains(string) {
                        let response: Result<Response<Body>, String>  = Ok(
                            Response::builder()
                                .status(StatusCode::OK)
                                .body(Body::from("Yes".to_string()))
                                .unwrap());
                        return response;
                    }
                }
                return Ok(
                    Response::builder()
                        .status(StatusCode::NOT_FOUND)
                        .body(Body::from("Not found"))
                        .unwrap());
            })
        );
        conn.await;
    }
}

Currently, I'm getting the error:

cannot move out of `strings_to_match`, a captured variable in an `FnMut` closure

move out of `strings_to_match` occurs hererustc(E0507)
main.rs(11, 9): captured outer variable
main.rs(18, 41): move out of `strings_to_match` occurs here
main.rs(19, 32): move occurs because `strings_to_match` has type `std::sync::Arc<std::vec::Vec<&str>>`, which does not implement the `Copy` trait
main.rs(19, 32): move occurs due to use in generator

I understand the error in practice: I have one strings_to_match, so I can't use it in the loop, because the previous iteration will take ownership. However, abstractly, this doesn't make sense to me. strings_to_match is an Arc, which impls Clone, so there should be an easy way to create a clone for every iteration right? I understand there is a difference between Copy and Clone in practice. Is my understanding correct? Is there an easy thing that I'm missing?

I've also tried:

  1. Removing the "move" doesn't help, as then it complains that it returns a value referencing req. This makes sense, since the async will return a Future, which references req. In that case, it makes sense that "move" is required to get ownership of req.
  2. I tried adding "let strings_to_match = strings_to_match.clone()" in both the while loop and inside the async closure. Both did not work, as to my understanding I'm still using the strings_to_match reference when calling clone.
  3. Following this, I tried calling Arc::clone on strings_to_match. This also didn't work for the same reason as above.

Are my understandings of the above correct?

For more context on what I'm trying to do, I'm trying to implement a TLS server with client cert verification that does extra handling of the client cert. I couldn't find any well maintained crates, so I rolled my own.

1 Like

To fix this you must first clone in the while loop, move that into the closure, and then clone in the closure and move that clone into the async block. The reason for this is that the closure passed to service_fn will be called multiple times, so even if you give the closure ownership of the arc, it can't give its only one away to the async block without cloning.

Remember, "async closure" is really an ordinary closure returning an async block.

let strings_to_match = strings_to_match.clone();
let conn = http.serve_connection(
    service_fn(move |req| {
        let strings_to_match = strings_to_match.clone();
        async move {
            ... 
        } 
    })
);
6 Likes

hah, I was just about to post almost the exact same question! Had a work-in-progress playground ready to go and all :wink: Thanks for saving the day @alice!

1 Like

Would it be possible to have a macro abstract over that pattern? something that could be called like:

async_clone_fn!(strings_to_match, |req| { ... })

I could try to do the heavy lifting and learn more about macros - just wondering if I'm barking up the wrong tree before I spend a day on that :sweat_smile:

My use case is I have a database connection pool which needs to be passed down to async handlers (specifically the type that gets passed to warp's and_then), and so I need to do this double-clone thing at each call site (or at least that's the only way I got it to compile so far!).

Yes, you can make such a macro like this.

macro_rules! async_clone_fn {
    ($share:ident, move |$($arg:ident),*| async { $($tok:tt)* }) => {
        {
            let $share = $share.clone();
            move |$($arg),*| {
                let $share = $share.clone();
                async move {
                    $($tok)*
                }
            }
        }
    };
}

Edit: or to allow sharing multiple values

3 Likes

Awesome, thanks again! Also a great concise example to learn about macros :slight_smile:

Is it necessary to explicitly have the async and move? I tried removing it from the macro declaration and invocation and it still compiles: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=b43082da40ac012deb0a14b6179b830f

No, since they are literals in the macro pattern, they don't have any meaning. They are necessary on the actual closure and async block inside the macro though.

1 Like

Thanks so much @alice, that was a very simple fix. Putting it into a macro as suggested by dakom is also a great idea.

In the interest of further understanding: why do we need the second clone inside the closure?

  1. You clone inside the while loop, so you get one instance per iteration
  2. You move this one instance into the closure
  3. Inside the closure, you clone again from the instance you cloned in the while loop
  4. You move your clone into the async block

Why is step 3 and 4 not redundant? You've already moved an instance of strings_to_match into the closure, why do you need to clone it before you move it into the async block, if there is only one async block per closure? Why can't you simply give the async block the same instance you moved into the closure?

I think your explanation here explains it:

The reason for this is that the closure passed to service_fn will be called multiple times, so even if you give the closure ownership of the arc, it can't give its only one away to the async block without cloning.

To me, you're saying that you need one instance per closure, but I'm trying to reference the book in chapter 13 and chapter 16, and it doesn't seem to be the case. What am I missing?

For reference, commenting out the second clone gives me: cannot move out of `strings_to_match rustc(E0507)`, a captured variable in an `FnMut` closure

This doesn't make sense to me, I moved strings_to_match into the async block, why does it constitute moving out if I don't explicitly give the async block its own instance?

Thanks again, you've probably saved me a few more hours :slight_smile:.

If it was possible to make it work without a clone in the closure, then the async block that the closure returns must somehow contain a reference into the closure, but closures are not able to return references into themselves.

This is not the case. Every time you call the closure, a new instance of the future produced by that async block is created.

If the closure gives its own strings_to_match away, it doesn't have one to give away the next time the closure is called. It constitutes moving out, because that's the only way the async block can be given access — to move something into it.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.