[Solved] Understanding why a closure requires 'static lifetime

help
1

Hi,

I'm trying to combine Rust with Python, in order to speed up a tree search algorithm I'm playing around with.

The outer program is a python script, which calls into a PyO3 wrapper around a Rust tree search.
This works well so far, but now I'm trying to occasionally evaluate a Python function within the Rust tree search and hitting the limits of my Rust knowledge.

I'm getting this lifetime error and am failing to understand why it thinks the lifetime should be 'static and how to fix it:

error[E0621]: explicit lifetime required in the type of `pyfun`
  --> src/lib.rs:90:19
   |
88 |     fn test4(&mut self, pyfun: &PyObjectRef) -> PyResult<(u8)> {
   |                                ------------ help: add explicit lifetime `'static` to the type of `pyfun`: `&'static pyo3::PyObjectRef`
89 |         let closure = || -> u8 { pyfun.call0().unwrap().extract().unwrap() };
90 |         Ok(search(&closure))
   |                   ^^^^^^^^ lifetime `'static` required

Here's a minimal example of what I'm trying to run and what still works:

In my python script I'm importing my rust module as m, and then calling different test methods with just a lambda as my "evaluation function":

print(m.test1(lambda: 1))
print(m.test2(lambda: 2))
print(m.test3(lambda: 3))
print(m.test4(lambda: 4))

The test functions are implemented in Rust, getting more complex and 1-3 work perfectly, with 4 I'm getting the error above:

// Just call the passed python function directly -> works
fn test1(&mut self, pyfun: &PyObjectRef) -> PyResult<(u8)> {
    pyfun.call0()?.extract()
}             
     
// Put the evaluation into a closure, directly call the closure -> works                 
fn test2(&mut self, pyfun: &PyObjectRef) -> PyResult<(u8)> {
    let closure = || -> u8 { pyfun.call0().unwrap().extract().unwrap() };
    Ok(closure())
}            

// placeholder for my tree search that should later call into the python function                                
type EvalFn = Fn() -> (u8);                            
fn search(f: &EvalFn) -> u8 {
    1 // later I'd call f in here, but not necessary to trigger my issue                                    
} 

// Try calling search with a Rust closure -> works
fn test3(&mut self, pyfun: &PyObjectRef) -> PyResult<(u8)> {
    let closure = || -> u8 { 33 };
    Ok(search(&closure))
}

// Now what I'm actually trying to do: search with a Python evaluation function
// -> errors with lifetime `'static` required
fn test4(&mut self, pyfun: &PyObjectRef) -> PyResult<(u8)> {
    let closure = || -> u8 { pyfun.call0().unwrap().extract().unwrap() };
    Ok(search(&closure))
}

I don't understand why it thinks the lifetime needs to be 'static when I put the pyfun evaluation into a closure, I don't think it should be necessary.
But I'm not sure how to debug this or tell it otherwise?

Any help would be greatly appreciated!

2

I'm not sure, but perhaps it's the lifetime elision which just defaults to 'static? In that case, try either:

// placeholder for my tree search that should later call into the python function
type EvalFn = dyn Fn() -> u8;                            
fn search<'a>(f: &'a EvalFn) -> u8 {
    1 // later I'd call f in here, but not necessary to trigger my issue
}

Or turning it into a generic and getting rid of the references and dynamic dispatch:

// placeholder for my tree search that should later call into the python function
fn search(f: impl Fn() -> u8) -> u8 {
    1 // later I'd call f in here, but not necessary to trigger my issue
}

Or turning it into a function pointer:

// placeholder for my tree search that should later call into the python function
fn search(f: fn() -> u8) -> u8 {
    1 // later I'd call f in here, but not necessary to trigger my issue
}

Or even using dynamic dispatch + 'lifetime

// placeholder for my tree search that should later call into the python function
fn search<'a>(f: &dyn Fn() -> u8 + 'a) -> u8 {
    1 // later I'd call f in here, but not necessary to trigger my issue
}

Or... Some combination of lifetimes...
PS. I presented the ideas in the order I thought they were the most likely to work.

1 Like
3

This error is misleading. It's not as much about requiring a 'static, as it is about forbidding all other kinds of references.

In your case search says it wants a closure that outlives your function, and can be used even after your pyfun is destroyed. So you're not allowed to use pyfun inside the closure.

2 Likes
4

Thanks!

fn search(f: impl Fn() -> u8) -> u8 {

seems to work, but I'm not entirely sure why yet.

5

The bit I'm missing is why it wants a closure that outlives my function.

In my mental model after desugaring etc. there should be a struct that captures pyfun, that gets passed into search(), search returns, the capture struct falls out of scope, the test4 function returns and everything works.

This is clearly not what's happening, but I'm not sure what's different.

How would I say my search wants a Fn that only needs to live for as long as search is running?
Is that along the same lines of what @OptimisticPeach wrote?

I've tried to implement a "closure" in the way I thought it worked, and that behaves as I expected:

struct Closure<'a>(&'a PyObjectRef);
impl<'a> Closure<'a> {
    fn call(&self) -> u8 {                                 
        self.0.call0().unwrap().extract().unwrap()
    }                      
}                                                    
                                                                                                                      
fn searchc(c: &Closure) -> u8 {
    c.call()      
}
6

Fn() -> (u8), which is now called dyn Fn() -> (u8) is an object, and it uses dynamic dispatch. Rust automatically adds +'static requirement to these, as they're usually put in Box<dyn>. You don't want these, as they're unnecessary complication and only make things less efficient here.

impl Fn() -> (u8) or F where: F: Fn() -> (u8) is generic, and it expands to the specific code, as if your closure was inlined there. And I think it should figure out the right lifetime.

3 Likes
7

Oh I see! Now it all makes sense (and works as well), thanks a lot for your help!

8

Here's a bug for the unclear error message:

https://github.com/rust-lang/rust/issues/48467

1 Like