Help sending `F : FnMut + 'env` accross threads in a multithreaded iterator processing library

I have a library implementing parallel iterator processing, preserving ordering and generally being very generic and drop-in replacement for single-threaded iterator operations.

One feature missing in it, and making it not an exact replacement for standard operations is lack of support for borrowed values. A normal iterator can map over non-static items, my library currently can't because I need to send iterator items over channel, and send mapping function to worker threads. I am 99% sure that I could have most (all) of this as well in a similar way crossbeam has "scopped threads". My library can easily guarantee that there are no references outliving the given lifetime on the worker threads it spawned, making all of this safe, I think.

I have given it a quick shot but I got stuck on an intersection of transmutting things and generics.

I'll get back to it when I have some time, but in the meantime if anyone can give me any pointers and help, I would greatly appreciate, and then maybe unblock another person that might want to use it a bit faster.

You can't rely on Drop to wait for completion, unless you keep full control of the object that must be dropped. Once it's in the user's hands, mem::forget is safe. That's why the old std::thread::scoped had to go:
https://cglab.ca/~abeinges/blah/everyone-poops/

In the callback style of scoped threads, the user has no chance to escape the cleanup at the end of the scope. The callback runs under catch_unwind so panics don't escape either, then the scope waits for all outstanding work before letting the lifetime end.

Oh dear. I know this story. But if my struct ParalellMap<'env> has that 'env can a user forget arbitrarily still? Somehow I though that will hold it, and then inside that ParalellMap I can guarantee no surprises, I think.

That 'env prevents it moving outside of that lifetime's scope, but it can still be forgotten.

Oh, I see. That's why in scope in crossbeam - Rust the &Scope is provided to the user as a reference, so the API can guarantee it will not be forgotten. I think I get it now.

I guess I could still provide this functionality like this:

thread::scope(|s| {
   borrowed_iter.parallel_map_scoped(s, |v| /* ... */).collect()
}).unwrap();

?

or something similar. Basically the same thing as crossbeam do, or maybe even use crossbeam directly. Do you think this will work?

One thing that might prevent me from using crossbeam is that I wanted to support arbitrary threadpools (mostly for control of parallelism), but that's kind of another story and I can deal with it one way or another.

Yeah, it should be possible to make it safe with a callback and scope parameter like that.

1 Like

One last question that I have is - after I figured out the safety with scopes - how do technically sneak through Item and FnMut like they were : 'static. I just can't figure out the right invocation in the unsafe block to cast them to 'static before moving and then back after moving.

After what feels like the longest lifetime war with the compiler ever and having to annotate 99% of the code with lifetimes and other bounds, I've got it all working: Add support for scoped iterators · dpc/pariter@e3ab227 · GitHub . I decided to use crossbeam::scope directly to make my life easier, and possibly allow interoperability with any other scoped-thread using code.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.