Dyn Fn and threads

help
1

As discussed in another topic

  • I have code that constructs impl Fns by composing impl Fns
  • These are executed on multiple threads (via rayon)
  • I'm trying to replace these with dyn Fns.

This has led me to replace lots of impl Fn(A) -> B + Send + Sync with Arc<dyn Fn(A) -> B + Send + Sync>, which is seems to be going OK until I have to deal with a closure which borrows a large lookup table: putting this closure in an Arc requires that the table have 'static lifetime.

My gut reaction is to try to solve this with scoped threads, and I found build_scoped and spawn_handler in rayon, but I didn't manage to get it to work with those.

Putting rayon aside and trying to use standard scoped threads as a stepping stone on the way to the real solution, and throwing out most of my domain-specific noise, I end up with this minimal example (need_shared_thing and run_it reflect the structure of my real code ... which may well need improving, but that's the context in which I'm trying to make progress at the moment)

use std::sync::Arc;

type Shared = std::collections::HashMap<Input, Output>;
type Input = usize;
type Output = i32;

fn main() {
    let shared_thing: Shared = std::collections::HashMap::new();

    let _out: Output = std::thread::scope(|s| {
        s.spawn(|| run_it(Arc::new(need_shared_thing(&shared_thing))))
    }).join().unwrap();

}

fn need_shared_thing(shared: &Shared) -> impl Fn(Input) -> Output + '_ {
    |i| *shared.get(&i).unwrap()
}

fn run_it(_: Arc<dyn Fn(Input) -> Output>) -> Output { todo!() }

(playground) which gives this compilation error

   Compiling playground v0.0.1 (/playground)
error: lifetime may not live long enough
  --> src/main.rs:11:9
   |
10 |     let _out: Output = std::thread::scope(|s| {
   |                                            -- return type of closure is ScopedJoinHandle<'2, i32>
   |                                            |
   |                                            has type `&'1 Scope<'1, '_>`
11 |         s.spawn(|| run_it(Arc::new(need_shared_thing(&shared_thing))))
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`

and mapping the message's lifetimes to those described in the last section of this page would give

  • '1 -> 'scope
  • '2 -> 'env

I'm failing to spot why returning the underlined value would require that 'scope outlive 'env. Probably because I've been working on the broader problem for too long by now, and I'm worried that I took a wrong turn somewhere along the way and might well be barking up the wrong tree altogether.

2

There are two issues:

  • You try to take the ScopedJoinHandle (the result of Scope::spawn()) outside of the thread::scope() closure. This can't work, since all of the threads must be joined before thread::scope() returns. (That's why the ScopedJoinHandle has an upper bound of 'scope.) Instead, you should join() the ScopedJoinHandle inside the thread::scope() closure.

  • In the signature of run_it(), Arc<dyn Fn(Input) -> Output> implicitly turns into Arc<dyn Fn(Input) -> Output + 'static>, since it's not behind a reference. To take a shorter-lived dyn Fn, you should write Arc<dyn Fn(Input) -> Output + '_> to accept any lifetime.

Fixing these leads to (Rust Playground):

use std::sync::Arc;

type Shared = std::collections::HashMap<Input, Output>;
type Input = usize;
type Output = i32;

fn main() {
    let shared_thing: Shared = std::collections::HashMap::new();

    let _out: Output = std::thread::scope(|s| {
        s.spawn(|| run_it(Arc::new(need_shared_thing(&shared_thing))))
            .join()
            .unwrap()
    });
}

fn need_shared_thing(shared: &Shared) -> impl Fn(Input) -> Output + '_ {
    |i| *shared.get(&i).unwrap()
}

fn run_it(_: Arc<dyn Fn(Input) -> Output + '_>) -> Output {
    todo!()
}
2 Likes
3

Thanks, that's very clear and very helpful.

Hmm, this is the tough one. I very vaguely recall having come across this tangentially somewhere. Where can I read more about this feature?

I omitted this lifetime in a signature elsewhere in my code, and tracking that down in a nontrivial context took quite some time. I wonder how long it would have taken me to spot it, if your answer handn't been fresh in my mind.

4

The exact rules are covered by the Rust Reference as default trait object lifetimes. The important part is that without an explicit + 'a lifetime annotation, trait objects behind references receive the lifetime of the reference, whereas owned trait objects (e.g., within a Box or Arc) receive either the 'static lifetime in signatures or an inferred lifetime in expressions.

1 Like
5

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.