I want to send work to a worker thread or threadpool that maintains an expensive-to-create thread-local resource. Specifically in my case, an ldap3::LdapConn. However, I believe this need could generalize to other use cases.
The work should be a FnOnce(T) -> R, where T is the resource under remote management, and R is the return value of the work.
Obviously, R has to be Send and 'static, because the threadpool doesn't know how long that value will be used when it is sent back to the controlling thread.
But, by the time that the remote returns control to the caller, F will never be used again. Why does F have to be static? We as the developers know that it is only required that the references that F captures live until the end of the call. crossbeam maintains a concept of scoped threads, but it joins the thread after the call is complete. I just want to keep using the same thread over and over.
To make this more concrete, here is the code I'm working with:
I actually have RemoteResource working now, provided that I constrain F to be 'static, but this is rough limitation.
After sending the closure to the thread; If there is a panic in the main thread then the referenced data may have been freed but the thread would treat it otherwise.
The controlling thread is blocked until the remote thread returns. How could it panic in the meantime?
Also, assume there is a way for that happen, wouldn't the same jeopardy apply to the references in closures sent to the scoped threads API of a package like Crossbeam?
This isn't required to be true of the threads in the standard library. It's straight-forward to have the main thread spawn thread A which spawns thread B. When thread A exits, thread B can keep running. If it referenced data from a stack frame of A, that would be memory unsafety.
Libraries like crossbeam or scoped-threadpool have APIs that statically prevent child threads from outliving the parent thread. This makes them both more capable and less capable than the standard library.
Thanks for your reply, Shep. I always appreciate your erudite answers to my novice questions, even though I feel bad when I can tell that it's something you've repeated many times in the past. Your patience is awe-inspiring!
I think I was misunderstanding the apis of the scoped threadpool packages. I thought they were creating a new thread for each task that was spawned on them. But that's incorrect. They simply guarantee that the threads don't outlive the scopes in which they're created.
In the end, scoped_threadpool didn't work because the execute method is &mut self, but I wanted to share the threadpool to an entire other pool of workers, so the pool couldn't be mutably shared. crossbeam_util provides scoped threads, but not a scoped threadpool. A github issue on Crossbeam ended with a recommendation to use rayon for scoped threadpools, and that API solved my problem well. (https://github.com/crossbeam-rs/crossbeam/issues/64)