`scope` variation with harder lifetimes cannot compile

[Ddystopia]

Hello, I'm writing thread::scope equivalent for threadx, and having some problems with lifetimes. Here is a small example of what it basically looks like. If remove thread from spawn arguments, it becomes pretty much the same as in std and it works. But in that state it does not compile. If change &mut thread to &thread it will not compile either, but use case is more understandable with &mut in my opinion, so I left it as it is.

My guess is that borrow checker is thinking that scope captured in closure, that is getting stored in thread, might be still referenced by the end of scope. But even if &mut is changed to & and body of Scope::spawn replaced with todo, error persists - maybe some interior mutability concerns? My guess is that I don't know how to express that I'll clean everything with lifetimes.

Do you have any thoughts how to get stuff like that to compile? I'm assuming it will involve unsafe code to tackle lifetimes, I'm absolutely fine with it - I'll ensure all safety conditions, just need to first express lifetimes right.

use std::marker::PhantomData;

struct Thread<F> {
    closure: Option<F>,
}
struct ThreadHandle<'a, F> {
    thread: &'a mut Thread<F>,
}
struct ScopeData<'scope> {
    data: PhantomData<&'scope ()>,
}
struct Scope<'scope, 'env: 'scope> {
    scope_data: ScopeData<'scope>,
    scope: PhantomData<&'scope mut &'scope ()>,
    env: PhantomData<&'env mut &'env ()>,
}
pub struct ScopedJoinHandle<'scope, 'params, F> {
    data: &'scope ScopeData<'scope>,
    handle: ThreadHandle<'params, F>,
}

impl<F> Thread<F> {
    const fn new() -> Self {
        Thread { closure: None }
    }
}

impl<'scope, 'env> Scope<'scope, 'env> {
    fn spawn<F>(
        &'scope self,
        thread: &'scope mut Thread<F>,
        f: F,
    ) -> ScopedJoinHandle<'scope, 'scope, F>
    where
        F: FnOnce() + Send + 'scope,
    {
        ScopedJoinHandle {
            data: &self.scope_data,
            handle: spawn(thread, f),
        }
    }
}

fn spawn<F>(thread: &mut Thread<F>, f: F) -> ThreadHandle<'_, F> {
    thread.closure = Some(f);
    ThreadHandle { thread }
}

fn scope<'env, F, P: 'static>(mut p: P, f: F)
where
    F: for<'scope> FnOnce(&'scope mut P, &'scope Scope<'scope, 'env>),
{
    let scope = Scope {
        scope_data: ScopeData { data: PhantomData },
        scope: PhantomData,
        env: PhantomData,
    };

    f(&mut p, &scope);

    drop(p);
}

fn main() {
    scope((Thread::new(), Thread::new()), |(t1, t2), s| {
        Scope::spawn(
            s, //
            t1,
            || _ = Scope::spawn(s, t2, || {}),
        );
    });
}

[Ddystopia]

Another possible variation of this code:

use std::marker::PhantomData;

struct Thread<F> {
    clojure: Option<F>,
}
struct ThreadHandle<'a, F> {
    thread: &'a mut Thread<F>,
}
struct ScopeData<'scope> {
    data: PhantomData<&'scope ()>,
}
struct Scope<'scope, 'env: 'scope> {
    scope_data: ScopeData<'scope>,
    scope: PhantomData<&'scope mut &'scope ()>,
    env: PhantomData<&'env mut &'env ()>,
}
pub struct ScopedJoinHandle<'scope, 'params, F> {
    data: &'scope ScopeData<'scope>,
    handle: ThreadHandle<'params, F>,
}

impl<F> Thread<F> {
    const fn new() -> Self {
        Thread {
            clojure: None,
        }
    }
}

impl<'scope, 'env> Scope<'scope, 'env> {
    fn spawn<'param: 'scope, F>(
        &'scope self,
        thread: &'param mut Thread<F>,
        f: F,
    ) -> ScopedJoinHandle<'scope, 'param, F>
    where
        F: FnOnce() + Send + 'scope,
    {
        ScopedJoinHandle {
            data: &self.scope_data,
            handle: spawn(thread, f),
        }
    }
}

fn spawn<F>(thread: &mut Thread<F>, f: F) -> ThreadHandle<'_, F> {
    thread.clojure = Some(f);
    ThreadHandle { thread }
}

fn scope<'env, F>(f: F)
where
    F: for<'scope> FnOnce(&'scope Scope<'scope, 'env>),
{
    let scope = Scope {
        scope_data: ScopeData { data: PhantomData },
        scope: PhantomData,
        env: PhantomData,
    };

    f(&scope);

    // assume unsafe clean up of borrowed data in `Thread` etc.
}

fn main() {
    let mut thread1 = Thread::new();
    let mut thread2 = Thread::new();

    scope(|s| {
        Scope::spawn(
            s, //
            &mut thread1,
            || _ = Scope::spawn(s, &mut thread2, || {}),
        );
    });
}

[kpreid]

I'm not sure what you're attempting to accomplish — in particular, the PhantomData stubs mean I don't know what the fields are actually intended to borrow — but these are almost certainly not what you want:

    scope: PhantomData<&'scope mut &'scope ()>,
    env: PhantomData<&'env mut &'env ()>,

An &mut T exclusive reference's referent, T, is invariant, which means that it cannot be coerced to involve a shorter lifetime. Because &'scope () and &'env () appear in this position, the lifetime parameters of Scope itself also become invariant. This means that it's not possible to treat a Scope<'a, 'b> from some outer scope as having a shorter 'a lifetime.

Furthermore, if you were borrowing any actual data with those mutable references, you'd find that you could not access it usefully — that the data becomes borrowed forever because it is borrowed for a lifetime it itself contains.

Then, putting these together, when you use a &'scope Scope<'scope, 'env> later, since Scope's parameters are invariant, you've created the actual borrowed-forever problem for that Scope.

I can't tell you what you should do instead, because your code is too abstract for me to know what you want the scope and env fields to actually do and so what lifetimes they should have, but you should remember this principle:

• &'a mut Foo<'a> is wrong; the two lifetimes must be different.
• &'a Foo<'a> might be wrong; check the variance of the 'a parameter of Foo.

Your problem might be solvable by just adjusting which lifetimes are used where, or you might need to introduce another lifetime parameter.

[Ddystopia]

Those two invariant lifetimes were taken from std code and there are comments why there are like that.

/// A scope to spawn scoped threads in.
///
/// See [`scope`] for details.
#[stable(feature = "scoped_threads", since = "1.63.0")]
pub struct Scope<'scope, 'env: 'scope> {
    data: Arc<ScopeData>,
    /// Invariance over 'scope, to make sure 'scope cannot shrink,
    /// which is necessary for soundness.
    ///
    /// Without invariance, this would compile fine but be unsound:
    ///
    /// ```compile_fail,E0373
    /// std::thread::scope(|s| {
    ///     s.spawn(|| {
    ///         let a = String::from("abcd");
    ///         s.spawn(|| println!("{a:?}")); // might run after `a` is dropped
    ///     });
    /// });
    /// ```
    scope: PhantomData<&'scope mut &'scope ()>,
    env: PhantomData<&'env mut &'env ()>,
}

[kpreid]

I see. I tried a few further things with your code, but I didn't manage to find something useful to change.