At a high-level, I’m trying to read and deserialize records from a file. Doing this serially is really slow, and so I’d like to do it in parallel. I have tried to create a minimal set of reproducing code, but couldn’t get it working in the Rust Playground, so forgive me for links to my project in GitHub.
My first attempt was using CpuPool via futures: https://github.com/wspeirs/logstore/blob/thread_pool/src/data_manager.rs#L79
When I try to compile this, I get the following error:
the type `[closure@src/data_manager.rs:93:44: 95:14 self:&mut data_manager::DataManager, loc:u64]` does not fulfill the required lifetime
I think I get this… basically
self doesn’t live forever (
'static) and because I’m passing it to a thread, that could live forever, that’s a problem. However, I haven’t a clue how to fix this. I’ve tried wrapping
Rc<RefCell<>> combo and
Arc<Mutext<>>, neither seem to work.
Next I tried using scoped-pool, as it addresses my exact issue. Using this approach, I run into borrowing issues: https://github.com/wspeirs/logstore/blob/thread_pool/src/data_manager.rs#L108
error[E0504]: cannot move `self` into closure because it is borrowed error[E0597]: `self_clone` does not live long enough error[E0597]: `loc` does not live long enough
I feel like I’m doing something fundamentally wrong here. I wouldn’t think it’d be that hard to issue multiple calls to
LogFile::get(...) and capture their results. How should I go about doing this? Thanks!