Backgroud
Assume we have a batch of transactions and the state view,which are immutable(&).Our target is to execute them parallelly,and the result of parallel execution should be consistent with executing them sequentially.
Current implementation
A recent academic paper have sloved above problem,whose implementation consists of a global scheduler module and a concurreny hashmap module.It use rayon as threadpool,each worker thread will continuously loop to process scheduler tasks came from scheduler.However,during the lifetime of scheduler task,it may block due to data dependency,thus waste cpu cycles.I want to dynamicly add/remove temp worker thread to use these cpu cycles.
Problem
I have tried to use scoped threads to implement my idea.
However,temp work thread also maybe block due to data dependency.
Theoretically, once data dependency has been resolve, blocked rayon thread can resume processing.
The Question is, if you have added temp worker thread before,you must wait until they joined(scoped thread's fork-join style),and unfortunately if the temp thread is also blocked because of data dependency,this unlucky rayon thread won't have chance to get scheduler task(waiting scoped thread join),which break the liveness guarantee of the paper's algorithm. This is why I have to use detached thread instead of scoped thread,I can inform detached thread to finish and don't need to wait it finished.
Some background can be found in this rayon issue.
Sorry for initial fuzzy problem statement,maybe this explaination can make everyone more clear about my problem.Thanks for everyone help me!
initial fuzzy description(useless)
In brief, my user case can be simplified to the following code demo.
#[derive(Clone)]
struct ParameterImpl;
trait Trait {
type Parameter: Send + Clone;
fn new(parameter: Self::Parameter) -> Self;
}
struct TraitImpl<'a> {
p: &'a ParameterImpl,
}
impl<'a> Trait for TraitImpl<'a> {
type Parameter = &'a ParameterImpl;
fn new(p: Self::Parameter) -> Self {
Self { p }
}
}
struct Foo<T: Trait> {
p: T::Parameter,
t: T,
}
impl<T: Trait> Foo<T> {
pub fn new(p: T::Parameter) -> Self {
let t = T::new(p.clone());
Self { p, t }
}
}
struct Bar<T: Trait> {
p: T::Parameter,
}
impl<T: Trait + 'static> Bar<T> {
pub fn new(p: T::Parameter) -> Self {
Self { p }
}
pub fn work(&self) {
let parameter = self.p.clone();
// I guarantee this thread will return before parameter dropped.(via some sync primitives)
let _ = std::thread::spawn(move || {
let _bar = Bar::<T>::new(parameter);
});
}
}
fn invoke(p: &ParameterImpl) {
let bar = Bar::<TraitImpl>::new(p);
bar.work();
}
fn main() {
let p = ParameterImpl;
invoke(&p);
}
Sadly, compiler tell me that it can't build for me because:
error[E0521]: borrowed data escapes outside of function
--> src/main.rs:47:15
|
46 | fn invoke(p: &ParameterImpl) {
| - - let's call the lifetime of this reference `'1`
| |
| `p` is a reference that is only valid in the function body
47 | let bar = Bar::<TraitImpl>::new(p);
| ^^^^^^^^^^^^^^^^^^^^^^^^
| |
| `p` escapes the function body here
| argument requires that `'1` must outlive `'static`
For more information about this error, try `rustc --explain E0521`.
error: could not compile `hello_rust` due to previous error
For some reason,I can't use scoped thread,which can borrow non-static data.
I think the trait bound T: Trait + 'static
on Bar
is not right.
How can I convince compiler that my code is right(the borrowed data can be safely use) , or any other way to impl this idea?