Idiomatic Outer Scope Inner Container Borrow

Is their an idiomatic way to achieve the following?

I am using an external API that uses FFI with the following interface.

Let these be two objects:
InOne
InTwo

InOne.getInTwo() - returns InTwo but as a borrow from InOne

I have a need to wrap InTwo into a RefCell to pass to other functions later. Also I have other to attach to InTwo.

I know the following works:

{ // outer scope
   let a = InOne::new();
   let b = RefCell::new(InOne.getInTwo());
   // call functions with using b
}

However, in setting up InOne.getInTwo() their is some redundant setup code I would like to extract out. The following is the interface I would prefer.

{ // outer scope
   let a = build_in_two();
   // ...
}

However, this is not possible as the internal InOne that InTwo borrows from would not outlive the scope. The only way I can think of achieving this is to return both.

{ // outer scope
   let a, b = build_in_one_in_two();
   // ...
}

However, I am running into issues from the borrow checker as obviously inside of "build_in_one_in_two", I am borrowing from a to build b. This presents an issue when moving a out of context. However, I am moving both a and its referrer out of context (up a scope) so I would believe this to be a safe operation. However, is this possible in Rust or should I result to a macro to cut down on building redundancy?

So you want InTwo, which cannot outlive the InOne instance it was created from, to be 'static (e.g., to be able to freely move it around).

You can:

  1. Use ::rental;

  2. Use ::owning_ref;

  3. Change the API to enable creating a InTwo from something like a Arc<InOne>, and instead of storing a &'a InOne, in your InTwo wrapper, you store that Arc<InOne>. This way the InOne will stay alive as long as InTwo is, while also providing a non 'a-bounded InTwo.

2 Likes

Thanks. I will look more into these over the next couple of days. Both seem to fit the use case I am looking for. "rental" seems slightly better as I will be wanting to store other objects in a struct with these resources.

I am now running into another tricky error using this method (lifetimes).

error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting r
equirements
  --> src/python.rs:21:46
   |
21 |     let pyi = or.map(|gil| &RefCell::new(gil.python()));
   |                                              ^^^^^^
   |
note: first, the lifetime cannot outlive the anonymous lifetime #2 defined on the b
ody at 21:22...
  --> src/python.rs:21:22
   |
21 |     let pyi = or.map(|gil| &RefCell::new(gil.python()));
   |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that reference does not outlive borrowed content
  --> src/python.rs:21:42
   |
21 |     let pyi = or.map(|gil| &RefCell::new(gil.python()));
   |                                          ^^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the functio
n body at 17:21...
  --> src/python.rs:17:21
   |
17 | pub fn build_py_ref<'a>() -> PythonInstance<'a> {
   |                     ^^
   = note: ...so that the expression is assignable:
           expected owning_ref::OwningRef<_, std::cell::RefCell<pyo3::python::Pytho
n<'a>>>
              found owning_ref::OwningRef<_, std::cell::RefCell<pyo3::python::Pytho
n<'_>>>

With code:

type PythonInstance<'a> = OwningRef<Box<GILGuard>, RefCell<Python<'a>>>;
pub fn build_py_ref<'a>() -> PythonInstance<'a> {
    let gil = Box::new(Python::acquire_gil());
    let or = OwningRef::new(gil);

    let pyi = or.map(|gil| &RefCell::new(gil.python()));
    pyi
}

This is due to the need for Python to have an internal lifetime related to the GIL.

Any ideas on how to best get rid of this? I have to confess I have yet to master the more complex aspects of lifetimes.

Yes, you have an unbounded lifetime parameter 'a in the return type, which is a code smell (it might as well be 'static). OwningRef will not make the cut, so you could try with rental.

However, the whole code pattern doesn't look natural. What are you trying to accomplish? There may be better solutions for your end-term goal than what you are currently attempting to do.

For instance, although I don't really know if it will be relevant to you, you could go and define:

fn with_gil<R, F> (f: F) -> R
where
    for<'gil> F : FnOnce(Python<'gil>) -> R,
{
    let gil = GILGuard::acquire();
    f(gil.python())
}

so that you could acquire the GIL doing:

let x = with_gil(|py| {
    // use py here
    42
}); // GIL released here

I am trying to accomplish the following.

let pythons = (0..n).map(|_| build_py()).collect();
// other things will be assigned to specific pythons
// multithreaded code [to be done at a future data]
{
   data_generation_func(pythons[i].py);
}

In am doing this in a high-performance setting so creating and destroying Python instances is not good. I want to eliminate setup and tear down time.

So there is no other way of accomplishing the following? I would hate to have to use a hacky macro to accomplish this.

The Python interpreter does not support multi-threading, so if you need to use it to generate a bunch of Python objects requring a Python<'gil> to be constructed, you cannot speed this up by multi-threading.

You can try with:

let gil = GILGuard::acquire();
let pythons: Vec<_> = 
    (0 .. n)
        .map(|_| gil.python())
        .collect()
;
::crossbeam::thread::scope(|scope| {
    pythons
        .iter()
        .for_each(|py| {
            scope.spawn(move /* py */ |_| {
                drop(py); // use py for anything
            })
        });
}).expect("Some thread panicked");

but you will have errors complaining about Python<'_> not being Send[able to other threads] because GILGuard is not Sync (i.e., a GILGuard cannot be accessed from multiple threads).

Does constructing multiple pythons from multiple GILGuards not construct multiple processes? I know that Python multithreading is not the best which is why I thought this was the best method to work with.

I don't know how pyo3 handles the GIL internally, but I imagine that instead of spawn multiple processes, but it will lock the current thread instead; that is, if we change the above code to:

::crossbeam::thread::scope(|scope| {
    (0 .. n)
        .for_each(|_| {
            scope.spawn(|_| {
                let py = GILGuard::acquire().python();
                drop(py); // use py for anything
            });
        });
}).expect("Some thread panicked");

this will should compile and run without problems, but I expect most threads to be kept locked at the GILGuard::acquire waiting for one of them to complete, and only then will another be able to run, etc.


I really don't know about multiprocessing Python from within Rust (you could spawn multiple ::std::process::Commands and then extract results from their repective stdouts, I guess...), so let's wait for someone else (cc @konstin @kngwyu ?) having actually worked with pyo3 or similar to help you with this issue :wink:

Sorry I didn't the whole thread in detail, but what I can say is Python locks all threads and you should use allow_threads to get the true parallelism.