Python Rust Interop

Is @bheisler 's blog post from 2017 https://bheisler.github.io/post/calling-rust-in-python/ the current state of the art in Rust <-> Python interop?

Or is there some other crate to look into?

1 Like

I would be interested in the answer to this myself. Since I wrote that, I’ve gotten a job at a Python shop and I’ve been wishing for a good excuse to introduce Rust. It’d be easier to justify if integrating the two were super easy to do. The method in that post is pretty effort-intensive.

2 Likes

AFAIK PyO3 is the crate of choice for this at the moment.

2 Likes

PyO3 looks very interesting.

I see an example for compiling Rust into a Lib that Python can import.

I also see an example of constructing a Python Interpreter inside the Rust runtime.

Is there a way to combine the two and (1) create a Python interpreter inside the rust runtime and (2) pass a Rust function to the Python runtime?

I haven’t looked into this very much, but there’s PyOxidizer which claims to help with both these use cases.

I’m using PyO3 to create a Blender plugin that is implemented in Rust and I love it. PyO3 makes things pretty straight forward, and while some code can be fairly verbose when trying to execute Python from Rust, it is easy to do.

I’m pretty sure that is possible. I have not done that exactly, but from everything I’ve seen there is nothing stopping you.

I’m going to look into PyO3 to integrate rust into some of my python packages. It looks pretty useful, thanks to the OP for bring this up.

Does anyone know if native python modules written in rust would be subject to the GIL when called from python? For instance, writing a module with functions to do thread-based multiprocessing - as this is clunky in python.

You can just use threads in Rust like you are used to, even if called from Python. Of course PyO3 will I guess keep the GIL as long as your function is running, but inside your function you are free to do whatever.

2 Likes

Awesome, thank you!

PyO3 also has a Python::allow_threads function that allows you to pass in a closure that actually will release the GIL while the closure is running which allows other Python threads to run at the same time.

I am wary of PyO3. The last time I seriously considered using it (a little under 2 years ago), it felt like basically every feature I looked at fell into one of two categories:

  • Either it already existed in rust_cpython (which PyO3 is a fork of)…
  • …or I could easily come up with a way to invoke UB with it.

After reporting the issues I found, I even had to work to convince the author that the UB was a problem!

In the time since then, a major shitstorm happened with unsafe code in the actix crate by the same author. I would like to think he has learned his lesson, but for me, the damage has been done; I have difficulty bringing myself to use the crate without heavy review. (and there’s simply too much of it for me to review!)

And even looking at it today, I see things that make sirens go off in my head:

fn try_from_mut<V: Into<&'v PyAny>>(
    value: V
) -> Result<&'v mut Self, PyDowncastError>

Surely this can be called twice on a &'v PyAny to get aliasing mutable references to Self?

impl<'a, T> std::convert::From<&'a T> for PyObject
where
    T: AsPyPointer;

Where did the lifetime go? Moreover, why on Earth does this even exist? I can only assume there’s some way to get a &'a T back out of the PyObject, thereby allowing a use-after-free? (Edit: I just noticed the AsPyPointer bound. That might make this less bad)

self.inner
    .push_back(unsafe { mem::MaybeUninit::uninit().assume_init() });

mem::MaybeUninit::uninit().assume_init() is explicitly documented as UB even for types like u32 that have no invalid bit patterns.


I’m not certain that I have the time and energy to hunt down and report examples of all the ways that one can still invoke UB using this crate. I’d feel much more comfortable just contributing to rust_cpython to add any missing features.

5 Likes

To be fair, the author of actix is not involved in this library any longer. It is in fact a collaboration between several active contributors, which in my book is a very good sign for long term maintenance.

@konstin @kngwyu & co. might be around to comment on your concerns.

2 Likes

Yeah, I agree with @ExpHP that there still remains lots of unsafe code in PyO3, which sometimes irritates me.
Since PyO3 aims to imitate the behavior of Python’s GIL, I know we cannot avoid some kind of unsafety.
But I’m still investigating a better design with limited my resources.

This is all I can say :slight_smile:
Thank you @rkarp @zeroexcuses @zicklag @amsesk for your positive reaction to PyO3.

3 Likes