Struggling a bit with ownership and closures

help
1

I've been working at this for about two weeks now and I feel like I've hit that point where I'm going in circles with no progress. So, here goes...

I decided I wanted to work on a project in Rust to learn it; that sort of thing works better for me, and I've settled on a client/server that sends and receives HTTP requests. Hyper seemed like a decent choice for that as it covers both sending requests and handling requests.

I have gone through the basics with Hyper and gotten the basic server running. I've started now trying to expand this into a dynamic dispatcher/router of sorts, the idea being that at the start of the program we load up some routes, and the dynamic dispatcher will perform some (currently very arbitrary) action. All fine.

I decided I wanted to break the dispatcher into its own module to play a bit with Rust's module system, and built a struct (Dispatcher) that will handle storing a route-to-action map. I've also added a dispatcher function that would do the work of checking the Request against the route-to-action map and would, eventually, do the action (or hand it off to be done elsewhere - whatever.)

Here is where I start running into issues. If I try the following:

use crate::dispatcher::dispatcher::{Dispatcher, dispatch_request} 
// my Dispatcher struct and dispatcher function

let dispatcher = Dispatcher::new();
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let server = Server::bind(&addr).serve
(
    make_service_fn(|_conn| async move
    {
        Ok::<_, Infallible>
        (
            service_fn
            (
                |req| async move {dispatch_request(req, dispatcher)}
            )
        )
    })
);

Apologies for the (\{ characters on their own line, I just find it easier to read closures if I blow them up like that.

Here's Dispatcher, if it matters:

#[derive(Clone)]
pub struct Dispatcher
{ 
    route_map: HashMap<String, String>,
}

impl Dispatcher 
{
    // pub fn new(prologue_routes: HashMap<String, String>) -> Self
    pub fn new() -> Self
    {
        Dispatcher { route_map: HashMap::new() }
    }
}

If I build it (or just using the rust-analyzer VS code plugin) I get cannot move out of 'dispatcher', a captured variable in an 'FnMut' closure, with the further notes move occurs because dispatcher has type 'Dispatcher', which does not implement the 'Copy' trait and move occurs due to use in generator. For reference, the actual compiler errors are:

error[E0507]: cannot move out of `dispatcher`, a captured variable in an `FnMut` closure
  --> src/main.rs:64:51
38 |         let dispatcher = Dispatcher::new(prologue_routes);
   |             ---------- captured outer variable
...
64 |                             |req| async move {dispatch_request(req, dispatcher)}
   |                                              ^^^^^^^^^^^^^^^^^^^^^^^----------^^
   |                                              |                      |
   |                                              |                      move occurs because `dispatcher` has type `Dispatcher`, which does not implement the `Copy` trait
   |                                              |                      move occurs due to use in generator
                                                  move out of `dispatcher` occurs here

I can kinda see that there would be lifetime issues here - if I understand the operation of Hyper correctly, the make_service_fn function takes a closure that is used to generate additional closures dynamically, one for each inbound HTTP request. So any data referenced in a variable that is in the innermost scope would go out of scope at the end of that closure, correct? If so, I can see that just moving a one-off reference to a variable into the innermost closure would not go well.

However - I'm still not sure how to solve the problem, or if what I'm trying can even be done given the mechanisms involved - nor am I positive that the error message matches the lifetime issues that I am guessing at. I've been hunting up and down trying to come to a better understanding of the mechanisms at play and clearly I've been missing something.

The closest I've found to someone with this exact same problem is here: https://users.rust-lang.org/t/solved-channel-in-a-loop-in-a-thread-borrowed-value-does-not-live-long-enough/26733/4. Yandros' answer partway down suggests wrapping the Dispatcher in an Arc and then cloning it in. On the surface seems to make sense...but doing so brings the exact same cannot move out of 'dispatcher' errors:

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let server = Server::bind(&addr).serve
(
    make_service_fn(|_conn| async move
    {
        let arc_dispatcher2 = arc_dispatcher.clone();
        Ok::<_, Infallible>
        (
            service_fn
            (
                |req| async move
                {
                    let arc_dispatcher3 = arc_dispatcher2.clone();
                    dispatch_request(req, arc_dispatcher3)
                }
            )
        )
    })
);

Trying with only one or the other of the arc_dispatcher.clone() results in no difference. Any attempt to move a var declared outside into any of the interior closures results in the cannot move error. And as I start to think about it - why would an Arc behave differently? The arc_dispatcher variable is still a variable subject to ownership and borrowing rules - so by rights when I reference arc_dispatcher in the make_service_fn closure it should get moved before the clone function is called. Or am I misunderstanding something here?

Sidenote: Removing the move keyword simply changes the error to a lifetime one - that the arc_dispatcher variable (or plain dispatcher in example 1) does not live long enough. Which maybe suggests my guess about lifetimes wasn't entirely wrong.

So at this point - I'm stuck. I guess I'm looking for help in fully understanding what, exactly, is happening here, and if my idea of passing in a struct to a dispatch function called by the Hyper service closure is even viable, or if I need to come up with some other solution entirely. Any help would be appreciated!

2

Put the clone outside the closure, otherwise you are still moving the original dispatcher inside the closure

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let arc_dispatcher2 = arc_dispatcher.clone();  // outside closure 
let server = Server::bind(&addr).serve
(
    make_service_fn(|_conn| async move
    {
        Ok::<_, Infallible>
        (
            service_fn
            (
                |req| async move
                {
                    let arc_dispatcher3 = arc_dispatcher2.clone(); // this clone probably isn't necessary
                    dispatch_request(req, arc_dispatcher3)
                }
            )
        )
    })
);
3 Likes
3

why would you want to clone the arc_dispatcher in async block , notice that the value has been move into async block,

1 Like
4

Apologies. I have tried that, I just forgot to mention it. The above code came from the first attempt after reviewing the thread I linked, where the poster showed doing it inside the closure. When that did not work, I added a clone outside the closure as shown in your code snippet, and encountered the same cannot move out of error.

I also attempted to use an intermediate clone in the outermost closure, and re-clone that on the innermost one:

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let arc_dispatcher1 = arc_dispatcher.clone();  // outside closure 
let server = Server::bind(&addr).serve
(
    make_service_fn(|_conn| async move
    {
        let arc_dispatcher2 = arc_dispatcher1.clone(); 
        Ok::<_, Infallible>
        (
            service_fn
            (
                |req| async move
                {
                    let arc_dispatcher3 = arc_dispatcher2.clone(); // this clone probably isn't necessary
                    dispatch_request(req, arc_dispatcher3)
                }
            )
        )
    })
);

I've also tried commenting out the intermediate block, and at your note that the inner clone is not necessary I removed it. All of them result in effectively the same error - the outer capture line updates to point to the outermost clone, but the rest is the same.

For me I guess the question is - why would moving a clone of an arc have any different side effects than moving the original? What does arc do that would make a clone of it special? Especially if the issue is to do with lifetimes:

  • Clone arc_dispatcher1 created outside of make_service_fn closure
  • arc_dispatcher1 is moved into make_service_fn closure's scope, arc_dispatcher2 is created
  • arc_dispatcher2 is moved into service_fn closure's scope
  • Once service_fn scope ends, the items moved into it would be destroyed - including arc_dispatcher2
  • Once make_service_fn scope ends, the items moved into it would be destroyed - including arc_dispatcher1

These closures are not FnOnce, so they can be called repeatedly - so the code in each scope layer would reference an item that had to be created in the enclosing scope, which may no longer be there the next time the code is called (as all of this is async). Or am I just barking up the wrong tree entirely?

5

Can you post the complete error. There's usually some context that can help

Oh! In that case you will also need to clone before the async block. Otherwise the closure will be inferred as FnOnce because you are moving out of arc_dispatcher2 into the async block.

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let arc_dispatcher2 = arc_dispatcher.clone();  // outside closure 
let server = Server::bind(&addr).serve
( // you may need to clone before this async move inside the closure
    make_service_fn(|_conn| async move
    {
        Ok::<_, Infallible>
        (
            service_fn
            ( // you may need to clone before this async move inside the closure
                |req| async move
                {
                    dispatch_request(req, arc_dispatcher2)
                }
            )
        )
    })
);
6

Certainly. From this block of code:

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let arc_dispatcher2 = arc_dispatcher.clone();
let server = Server::bind(&addr)
    .serve(
        make_service_fn(|_conn| async move
        {
            // let arc_dispatcher2 = arc_dispatcher1.clone();
            Ok::<_, Infallible>
            (
                service_fn
                (
                    |req| async move 
                    {
                        let arc_dispatcher3 = arc_dispatcher2.clone();
                        dispatch_request(req, arc_dispatcher3)
                    }
                )
            )
        })
);

I get the following error generated:

error[E0507]: cannot move out of `arc_dispatcher2`, a captured variable in an `FnMut` closure
  --> src/main.rs:66:29
   |
55 |           let arc_dispatcher2 = arc_dispatcher.clone();
   |               --------------- captured outer variable
...
66 | /                             {
67 | |                                 let arc_dispatcher3 = arc_dispatcher2.clone();
   | |                                                       ---------------
   | |                                                       |
   | |                                                       move occurs because `arc_dispatcher2` has type `Arc<Dispatcher>`, which does not implement the `Copy` trait
   | |                                                       move occurs due to use in generator
68 | |                                 dispatch_request(req, arc_dispatcher3)
69 | |                             }
   | |_____________________________^ move out of `arc_dispatcher2` occurs here

error[E0507]: cannot move out of `arc_dispatcher2`, a captured variable in an `FnMut` closure
  --> src/main.rs:59:17
   |
55 |             let arc_dispatcher2 = arc_dispatcher.clone();
   |                 --------------- captured outer variable
...
59 |   /                 {
60 |   |                     // let arc_dispatcher2 = arc_dispatcher1.clone();
61 |   |                     Ok::<_, Infallible>
62 |   |                     (
...    |
65 | / |                             |req| async move 
66 | | |                             {
67 | | |                                 let arc_dispatcher3 = arc_dispatcher2.clone();
68 | | |                                 dispatch_request(req, arc_dispatcher3)
69 | | |                             }
   | | |                             -
   | | |                             |
   | |_|_____________________________move occurs because `arc_dispatcher2` has type `Arc<Dispatcher>`, which does not implement the `Copy` trait
   |   |                             move occurs due to use in generator
70 |   |                         )
71 |   |                     )
72 |   |                 })
   |   |_________________^ move out of `arc_dispatcher2` occurs here

As for cloning outside the async block - I'm not sure what you mean? Currently arc_dispatcher2 is the result of cloning, so it's a clone before the async move into the make_service_fn closure - would I need another one? In terms of code, where would that go? The next thing after the outer clone is the function call whose first param is the closure.

I have tried an additional clone in the make_service_fn closure if that's what you're referring to - I've left it commented out in the example above. The errors stay the same - it just starts complaining about two moves (into make_service_fn as well as service_fn closures) instead of just the one.

7

I think you need need a clone that is inside the inner closure (which should be a move closure) but outside of its async move block.

service_fn(
    move |req| {
        let arc_dispatcher3 = arc_dispatcher2.clone();
        async move {
            dispatch_request(req, arc_dispatcher3)
        }
    }
)

You may also need to do the same thing for the outer closure and its async move block. (Playground)

3 Likes
8

Oh - interesting. I thought I had to mark the entire closure as async - that's some new learning there.

Also - that worked! I had to also clone the dispatcher into the outer closure as you suggested. Ignoring that left me with the move error hanging around on the outer block.

Finished code:

let arc_dispatcher = Arc::new(Dispatcher::new(prologue_routes));
let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
let arc_dispatcher1 = arc_dispatcher.clone();
let server = Server::bind(&addr).serve (
        make_service_fn( |_conn| 
        {
            let arc_dispatcher2 = arc_dispatcher1.clone();
            async move
            {
                Ok::<_, Infallible> (service_fn(move |req|
                {
                    let arc_dispatcher3 = arc_dispatcher2.clone();
                    async move 
                    {
                        dispatch_request(req, arc_dispatcher3)
                    }
                }))
            }
        })
    );

Thanks everyone for the help and doubly so to mbrubeck for the working solution. I'll be a while figuring out how that works but - I mean, it works. So that's neat.

9

so clone it before the move, lol

10

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.