Understanding lifetimes in parallel context

[lyuben-todorov]

Hi, I'm new to Rust so I'm sorry for the stupid question. I'm trying to do what this chapter is demonstrating how not to do.

Code:

// Start discovery routine
pub fn dsc(thread_pool: &ThreadPool, state: &mut State, neighbour_list:  &Vec<String>) {
    let node_list: Arc<Mutex<HashMap<String, Node>>> =
        Arc::new(Mutex::new(HashMap::new()));

    for host in neighbour_list {
        let nodes = Arc::clone(&node_list);
        thread_pool.execute(move || handshake(&host, nodes));
    }

    thread_pool.join();
    state.change_mode(Mode::WRK);
}

fn handshake(host: &String, node_list: Arc<Mutex<HashMap<String, Node>>>) {
    // do requests
}

Compile error:

error[E0621]: explicit lifetime required in the type of `neighbour_list`
  --> src/discovery.rs:16:21
   |
10 | pub fn dsc<'a>(thread_pool: &ThreadPool, state: &mut State, neighbour_list:  &Vec<String>) {
   |                                                                              ------------ help: add explicit lifetime `'static` to the type of `neighbour_list`: `&'static std::vec::Vec<std::string::String>`
...
16 |         thread_pool.execute(move || handshake(&host, nodes));
   |                     ^^^^^^^ lifetime `'static` required

I do understand it's not compiling because there are no lifetime guarantees for the contents of neighbour_list after the execute call. The calling thread could just exit at any time. However, I intend to wait for each job to finish with the thread_pool.join() call. I know about lifetimes, but I don't know how I can infer the them here. Or is this not the right approach at all?

[kornel]

Rust's borrow checker doesn't care what the code actually does. It only checks what code declares that it does. A call to threapool.join() doesn't change these declarations, so it doesn't do anything for borrow checking. Borrow checking operates on very general rules tied to scopes and declared lifetimes, without understanding any of the details.

So if execute says that 'static is required, this is all that matters. It means all temporary borrows are completely forbidden, period.

In safe Rust you can't sneak temporary references in any way into a 'static closure. You could do:

let host = host.clone(); // changes &String to String

to give each closure an owned string. Or change neighbour_list: &Vec<String> to neighbour_list: Vec<String> to reuse owned strings from the Vec.

With unsafe you could cast &String to &'static String at your own risk to trick execute into accepting it, and be careful no to "leak" the static reference out of the closure.

The best solution to all of this is to use a scoped threadpool, such as rayon::scope. That scoped threadpool is constructed to guarantee to always wait for all tasks in scope, and therefore can allow temporary borrows in the tasks.

[lyuben-todorov]

Thanks! This was a really helpful insight. I wanted to avoid using a bigger library such as rayon at first but I think it could be my only solution. I can't really afford to change &Vec<String> to Vec<String>, as I can't pass its ownership to fn dsc() (it exists in a control flow loop). In reality, my real problem is waiting for a pool of concurrent jobs to all finish, which is not at all trivial and I shouldn't expect it natively.

[kornel]

BTW: &Vec<T> is usually wasteful type to use, since it causes double indirection. Use &[T] whenever you don't pass ownership.