Help with multithreading and lifetimes. Can't figure out how to chunk up a variable without making it static

Attempting to multithread a simulation I'm working on, but struggling to make this work at the moment. A simple solution is to specify &'static mut self, But that seems unnecessary and causes other issues. Any help is appreciated.
Full Error Message:

error[E0759]: `self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
   --> src/main.rs:303:22
    |
286 |         &mut self,
    |         --------- this data with an anonymous lifetime `'_`...
...
303 |         let chunks = self.investor_vec.chunks_mut(chunk_size);//.chunks(chunk_size).collect();
    |                      ^^^^^^^^^^^^^^^^^ ...is used here...
...
306 |             handles.push(thread::spawn( move || {
    |                          ------------- ...and is required to live as long as `'static` here
    |
note: `'static` lifetime requirement introduced by this bound
   --> /Users/ewoolsey/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/mod.rs:649:15
    |
649 |     F: Send + 'static,
    |               ^^^^^^^

For more information about this error, try `rustc --explain E0759`.```

impl DemandSim {
    
    pub fn get_demand_and_update(
        &mut self,
        reserve_ratio: TimeVec,
        cd_price: TimeVec,
        usd_inflation: TimeVec,
        global_interest_rate: TimeVec,
    ) -> (f64, DateTime<Utc>) {

        self.time = self.time + self.settings.time_step;
        let current_time = self.time;

        let oppertunity_cost_index = get_oppertunity_cost_index(usd_inflation, global_interest_rate, current_time);
        let reserve_index = get_reserve_index(reserve_ratio, current_time);
        let volitility_index = get_volitility_index(cd_price, current_time);
        let inflation_index = get_inflation_index(cd_price, current_time);

        let mut total_demand = 0.0;
        let chunk_size = self.investor_vec.len() / NUM_THREADS;
        let chunks = self.investor_vec.chunks_mut(chunk_size);//.chunks(chunk_size).collect();
        let mut handles = vec![];
        for chunk in chunks {
            handles.push(thread::spawn( move || {
                let mut demand = 0.0;
                for investor in chunk {
                    demand += investor.get_demand_and_update(
                        oppertunity_cost_index,
                        reserve_index,
                        volitility_index,
                        inflation_index,
                        current_time
                    );
                }
                demand
            }));
        }
        for handle in handles {
            total_demand += handle.join().unwrap();
        }
        (total_demand, current_time)
    }
}

This kind of thing can be made to work using scoped thread as offered by the crossbeam crate or also soon in the standard library.


A more ergonomic alternative might be just to use parallel iterators offered by rayon which will mean you don’t have to do any chunking, bookkeeping or thread-spawning yourself, and instead could just to something like self.investor_vec.par_iter_mut().map(|investor| investor.get_demand_and_update(…)).sum().

(Rayon uses a work-stealing thread-pool; so you even save some overhead from spawning the threads; or overhead from idle threads, in case the workload can be unbalanced and some threads finish faster than others, which can be mitigated via the work-stealing. Of course feel free to benchmark yourself whether it’s on par, or perhaps even beneficial in your concrete use case.)

Thanks for the info! I am somehow running into the same problem using scoped threads... it's really weird. I'm sure I'm doing something stupid.

    pub fn get_demand_and_update(
        &mut self,
        reserve_ratio: TimeVec,
        cd_price: TimeVec,
        usd_inflation: TimeVec,
        global_interest_rate: TimeVec,
    ) -> (f64, DateTime<Utc>) {

        self.time = self.time + self.settings.time_step;
        let current_time = self.time;

        let oppertunity_cost_index = get_oppertunity_cost_index(usd_inflation, global_interest_rate, current_time);
        let reserve_index = get_reserve_index(reserve_ratio, current_time);
        let volitility_index = get_volitility_index(cd_price, current_time);
        let inflation_index = get_inflation_index(cd_price, current_time);

        let mut total_demand = 0.0;
        let chunk_size = self.investor_vec.len() / NUM_THREADS;
        let chunks = self.investor_vec.chunks_mut(chunk_size);

        thread::scope( |s| {    
            let mut handles = vec![];
            for chunk in chunks {
                handles.push(s.spawn( move || {
                    let mut demand = 0.0;
                    for investor in chunk {
                        demand += investor.get_demand_and_update(
                            oppertunity_cost_index,
                            reserve_index,
                            volitility_index,
                            inflation_index,
                            current_time
                        );
                    }
                    demand
                }));
            }
            for handle in handles {
                total_demand += handle.join().unwrap();
            }
        });
        (total_demand, current_time)
    }
}

error message:

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements
   --> src/main.rs:304:22
    |
304 |         let chunks = self.investor_vec.chunks_mut(chunk_size);
    |                      ^^^^^^^^^^^^^^^^^
    |
note: first, the lifetime cannot outlive the anonymous lifetime defined here...
   --> src/main.rs:287:9
    |
287 |         &mut self,
    |         ^^^^^^^^^
note: ...so that reference does not outlive borrowed content
   --> src/main.rs:304:22
    |
304 |         let chunks = self.investor_vec.chunks_mut(chunk_size);
    |                      ^^^^^^^^^^^^^^^^^
note: but, the lifetime must be valid for the anonymous lifetime #1 defined here...
   --> src/main.rs:306:24
    |
306 |           thread::scope( |s| {
    |  ________________________^
307 | |             let mut handles = vec![];
308 | |             for chunk in chunks {
309 | |                 handles.push(s.spawn( move || {
...   |
325 | |             }
326 | |         });
    | |_________^
note: ...so that the types are compatible
   --> src/main.rs:309:32
    |
309 |                 handles.push(s.spawn( move || {
    |                                ^^^^^
    = note: expected `&std::thread::Scope<'_, '_>`
               found `&std::thread::Scope<'_, '_>`

For more information about this error, try `rustc --explain E0495`.```

Aww dang it.. looks like the spam filter might have disappeared your reply after I fixed its formatting (make sure that the ```s are on their own line; and feel free to use the preview on the right side of the editor to check the code blocks before posting, (you might need to close any messages that might be displayed there, in order to see the preview)).

IDK why, this kind of thing happend to me a few times recently. Feel free to re-post (or wait until mods resolve the issue).

1 Like

Still getting errors even with scoped threads...
error message:

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements
   --> src/main.rs:304:22
    |
304 |         let chunks = self.investor_vec.chunks_mut(chunk_size);
    |                      ^^^^^^^^^^^^^^^^^
    |
note: first, the lifetime cannot outlive the anonymous lifetime defined here...
   --> src/main.rs:287:9
    |
287 |         &mut self,
    |         ^^^^^^^^^
note: ...so that reference does not outlive borrowed content
   --> src/main.rs:304:22
    |
304 |         let chunks = self.investor_vec.chunks_mut(chunk_size);
    |                      ^^^^^^^^^^^^^^^^^
note: but, the lifetime must be valid for the anonymous lifetime #1 defined here...
   --> src/main.rs:306:24
    |
306 |           thread::scope( |s| {
    |  ________________________^
307 | |             let mut handles = vec![];
308 | |             for chunk in chunks {
309 | |                 handles.push(s.spawn( move || {
...   |
325 | |             }
326 | |         });
    | |_________^
note: ...so that the types are compatible
   --> src/main.rs:309:32
    |
309 |                 handles.push(s.spawn( move || {
    |                                ^^^^^
    = note: expected `&std::thread::Scope<'_, '_>`
               found `&std::thread::Scope<'_, '_>`

and code:

    pub fn get_demand_and_update(
        &mut self,
        reserve_ratio: TimeVec,
        cd_price: TimeVec,
        usd_inflation: TimeVec,
        global_interest_rate: TimeVec,
    ) -> (f64, DateTime<Utc>) {

        self.time = self.time + self.settings.time_step;
        let current_time = self.time;

        let oppertunity_cost_index = get_oppertunity_cost_index(usd_inflation, global_interest_rate, current_time);
        let reserve_index = get_reserve_index(reserve_ratio, current_time);
        let volitility_index = get_volitility_index(cd_price, current_time);
        let inflation_index = get_inflation_index(cd_price, current_time);

        let mut total_demand = 0.0;
        // self.investor_vec.par_iter_mut().map(|investor| investor.get_demand_and_update(
        //                     oppertunity_cost_index,
        //                     reserve_index,
        //                     volitility_index,
        //                     inflation_index,
        //                     current_time
        // )).sum::<f64>();
        let chunk_size = self.investor_vec.len() / NUM_THREADS;
        let chunks = self.investor_vec.chunks_mut(chunk_size);

        thread::scope( |s| {    
            let mut handles = vec![];
            for chunk in chunks {
                handles.push(s.spawn( move || {
                    let mut demand = 0.0;
                    for investor in chunk {
                        demand += investor.get_demand_and_update(
                            oppertunity_cost_index,
                            reserve_index,
                            volitility_index,
                            inflation_index,
                            current_time
                        );
                    }
                    demand
                }));
            }
            for handle in handles {
                total_demand += handle.join().unwrap();
            }
        });
        (total_demand, current_time)
    }
}

It does seem like your rayon solution would work well! Really neat crate... Super clean. I kinda want to compare the speed of each method now though.

Before I continue to try understanding the issue, here’s some potentially interesting additional information you could provide / things you could try: are you using an up-to-date nightly compiler (rustc --version)? Is your code edition 2021? Does the same kind of error appear if you try the scoped threads from crossbeam instead of std?

edition = "2021"
Let me try crossbeam

Just used crossbeam instead and it works great... Very weird that std doesn't have the same functionality. Thanks for your help, you're a rockstar!

That’s a potential problem then that might be relevant to fix for the stabilization of the std::thread::scope API. What’s your nightly compiler version you are using (as returned by rustc +nightly --version)? Is it recent enough that the API works without needing feature(scoped_threads)? If now, maybe try updating, before we could conclude there’s an issue.

If you are using an up-to-date compiler, I would be interested in reducing / mocking up the code into a self-contained demonstration test case.

rustc 1.63.0-nightly (fee3a459d 2022-06-05)
I did need to manually add the feature

Then could you please update (rustup update) and try again? Because there are some recent changes that should hopefully make your code above work, and those got merged only about a week ago. If after updating your compiler version is new enough that the feature(...) no longer is necessary for scoped threats, then it should hopefully also compile successfully without that lifetime error.

I can confirm it is now compiling using thread::scope :slight_smile: Thank you so much for everything!

1 Like