Lifetime issue when generalizing a function

I am trying to use pyo3 to process some data through python (for scriptability reasons) and I hit an interesting issue with lifetimes. I want to make the Processor::run method’s return type generic so I can use it to process and return different types.

Code

extern crate pyo3;

use pyo3::prelude::*;

struct Processor {
    gil: GILGuard,
    python_function: PyObject,
}

impl Processor {
    fn new() -> Processor {
        let gil = Python::acquire_gil();
        let python_function = gil.python().None();
        Processor { gil, python_function }
    }

    fn run<'p, T: FromPyObject<'p>>(&'p self) -> PyResult<T> {
        let py = self.gil.python();
        let obj = self.python_function.call0(py)?;
        obj.extract(py)
    }
}

fn main() {
    let p = Processor::new();
    let processed_result: String = p.run().unwrap();
}

But I get a lifetime error:

error[E0597]: `obj` does not live long enough
  --> src/main.rs:20:9
   |
17 |     fn run<'p, T: FromPyObject<'p>>(&'p self) -> PyResult<T> {
   |            -- lifetime `'p` defined here
...
20 |         obj.extract(py)
   |         ^^^------------
   |         |
   |         borrowed value does not live long enough
   |         argument requires that `obj` is borrowed for `'p`
21 |     }
   |     - `obj` dropped here while still borrowed

When I saw that the object was borrowed after leaving the function I thought I could just clone it and return the clone to fix it, but that was not the case:

    fn run<'p, T: FromPyObject<'p> + Clone>(&'p self) -> PyResult<T> {
        let py = self.gil.python();
        let obj = self.python_function.call0(py)?;
        let res: T = obj.extract(py)?;
        Ok(res.clone())
    }

The error message is pretty much the same. My hunch is that it can be caused by the generic type possibly being a reference type but I’m not sure how to approach this problem.

Docs

pyo3 documentation

Thanks :slight_smile:

You cannot directly get what you want: gil.python() yields a RAII guard that releases the GIL when dropped, but it is required to be held by the generic FromPyObject::extract.

To circumvent this restriction, you have multiple solutions:

  • the one that always works is taking a closure that handles the result:

    impl Processor {
        fn run_with<'processor, T, Ret, F> (
            self: &'processor Self,
            f: F,
        ) -> PyResult<Ret>
        where
            T : FromPyObject<'processor>,
            F : FnOnce(T) -> Ret,
        {
            let gil_guard = self.gil.python(); 
            let obj = self.python_function.call0(gil_guard)?;
            obj.extract(gil_guard).map(f) // gil is only released *after* f
        }
    }
    
    fn example (processor: &'_ Processor) -> PyResult<()>
    {
        type T = ...;
        processor.run_with(|result: T| {
            // GIL is still held here
            println!("result = {:?}", result);
        })?;
    }
    
  • a more dirty solution is to force inlining the body of the function so that gil_guard is a local from the caller instead of the callee, which can be done by defining a macro instead of a function.

  • a cleaner form of the previous idea (making the gil be held by the caller) is to refacto a little bit your code:

    impl Processor {
        fn lock_gil (self: &'_ Self) -> GILLockedProcessor<'_>
        {
            LockedProcessor {
                processor: self,
                gil_guard: self.gil.python(),
            }
        }
    }
    
    #[Derive(Clone, Copy)]
    struct GILLockedProcessor<'gil> {
        processor: &'gil Processor,
        gil_guard: &'gil GILGuard,
    }
    
    impl<'gil> LockedProcessor<'gil> {
        fn run<T : FromPyObject<'gil>> (
            self: Self,
        ) -> PyResult<T>
        {
            let obj =
                self.processor
                    .python_function
                    .call0(self.gil_guard)?
            ;
            obj.extract(self.gil_guard)
        }
    
    fn example (processor: &'_ Processor) -> PyResult<()>
    {
        type T = ...;
        let result: T = processor.lock_gil().run()?; // gil is held as long as result lives
        println!("result = {:?}", result);
        Ok(())
    }
    
2 Likes

The thing I don’t understand about the first approach is why would it be necessary to have the gil in scope if the resulting PyObject is not tied to its lifetime in any way… result of call0 is an object with no lifetime annotations which means I can move it even outside of the gil scope. Same goes for the extract function. From its definition, I don’t see a reason why the result of it would be tied to a lifetime of any of its arguments.

fn extract<'p, D>(&'p self, py: Python) -> PyResult<D> where D: FromPyObject<'p>

The only thing I’m having trouble with is generalizing it. If I use a concrete type like for example String it works:

// compiles
fn run(&self) -> PyResult<String> {
    let py = self.gil.python();
    let obj = self.python_function.call0(py)?;
    obj.extract(py)
}

// doesn't compile
fn run<'p, T: FromPyObject<'p>>(&'p self) -> PyResult<T> {
    let py = self.gil.python();
    let obj = self.python_function.call0(py)?;
    obj.extract(py)
}

Ok, it seems I skimmed too quickly at pyo3 documentation, and that their lifetimes refer to the objects instead of the GIL, my bad :sweat_smile:

Let’s try again, this time more focused:

  1. start point: playground

    fn run<'processor, T> (
        self: &'processor Self,
    ) -> PyResult<T>
    where
        T : FromPyObject<'processor>,
    {
        let py = self.gil.python();
        let obj = self.python_function.call0(py)?;
        obj.extract(py)
    }
    
  2. Now, as you can see, given that I have made the effort to give actual names to lifetimes (not doing it is a very bad habit that is sadly still too present, even in the official Rust docs!), we can notice that we are using a 'processor lifetime for the PyObject conversion, instead of a 'object lifetime.

    • but where could possibly come such 'object lifetime? - you may ask.

    • well, it comes from borrowing a local variable, so it could have any lifetime, even a very short one !

    Now, there are two options: either we are borrowing data from the created object, in which case a closure is needed (c.f. my previous answer), or we are not, such as with T = String (the case in your example), and then we can fix the lifetime issue with the following function signature:

    fn run<T> (
        self: &'_ Self, // we don't care about this lifetime
    ) -> PyResult<T>
    where
        for</*all lifetimes*/ 'obj> T : FromPyObject<'obj>,
    {
        let py = self.gil.python();
        let obj = self.python_function.call0(py)?;
        obj.extract(py)
    }
    
1 Like

Thank you very much… I haven’t heard of HRTBs until now…

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.