Python Rust Interop

Is @bheisler 's blog post from 2017 https://bheisler.github.io/post/calling-rust-in-python/ the current state of the art in Rust <-> Python interop?

Or is there some other crate to look into?

1 Like

I would be interested in the answer to this myself. Since I wrote that, I’ve gotten a job at a Python shop and I’ve been wishing for a good excuse to introduce Rust. It’d be easier to justify if integrating the two were super easy to do. The method in that post is pretty effort-intensive.

2 Likes

AFAIK PyO3 is the crate of choice for this at the moment.

2 Likes

PyO3 looks very interesting.

I see an example for compiling Rust into a Lib that Python can import.

I also see an example of constructing a Python Interpreter inside the Rust runtime.

Is there a way to combine the two and (1) create a Python interpreter inside the rust runtime and (2) pass a Rust function to the Python runtime?

I haven’t looked into this very much, but there’s PyOxidizer which claims to help with both these use cases.

I’m using PyO3 to create a Blender plugin that is implemented in Rust and I love it. PyO3 makes things pretty straight forward, and while some code can be fairly verbose when trying to execute Python from Rust, it is easy to do.

I’m pretty sure that is possible. I have not done that exactly, but from everything I’ve seen there is nothing stopping you.

I’m going to look into PyO3 to integrate rust into some of my python packages. It looks pretty useful, thanks to the OP for bring this up.

Does anyone know if native python modules written in rust would be subject to the GIL when called from python? For instance, writing a module with functions to do thread-based multiprocessing - as this is clunky in python.

You can just use threads in Rust like you are used to, even if called from Python. Of course PyO3 will I guess keep the GIL as long as your function is running, but inside your function you are free to do whatever.

2 Likes

Awesome, thank you!

PyO3 also has a Python::allow_threads function that allows you to pass in a closure that actually will release the GIL while the closure is running which allows other Python threads to run at the same time.

I am wary of PyO3. The last time I seriously considered using it (a little under 2 years ago), it felt like basically every feature I looked at fell into one of two categories:

  • Either it already existed in rust_cpython (which PyO3 is a fork of)…
  • …or I could easily come up with a way to invoke UB with it.

After reporting the issues I found, I even had to work to convince the author that the UB was a problem!

In the time since then, a major shitstorm happened with unsafe code in the actix crate by the same author. I would like to think he has learned his lesson, but for me, the damage has been done; I have difficulty bringing myself to use the crate without heavy review. (and there’s simply too much of it for me to review!)

And even looking at it today, I see things that make sirens go off in my head:

fn try_from_mut<V: Into<&'v PyAny>>(
    value: V
) -> Result<&'v mut Self, PyDowncastError>

Surely this can be called twice on a &'v PyAny to get aliasing mutable references to Self?

impl<'a, T> std::convert::From<&'a T> for PyObject
where
    T: AsPyPointer;

Where did the lifetime go? Moreover, why on Earth does this even exist? I can only assume there’s some way to get a &'a T back out of the PyObject, thereby allowing a use-after-free? (Edit: I just noticed the AsPyPointer bound. That might make this less bad)

self.inner
    .push_back(unsafe { mem::MaybeUninit::uninit().assume_init() });

mem::MaybeUninit::uninit().assume_init() is explicitly documented as UB even for types like u32 that have no invalid bit patterns.


I’m not certain that I have the time and energy to hunt down and report examples of all the ways that one can still invoke UB using this crate. I’d feel much more comfortable just contributing to rust_cpython to add any missing features.

6 Likes

To be fair, the author of actix is not involved in this library any longer. It is in fact a collaboration between several active contributors, which in my book is a very good sign for long term maintenance.

@konstin @kngwyu & co. might be around to comment on your concerns.

3 Likes

Yeah, I agree with @ExpHP that there still remains lots of unsafe code in PyO3, which sometimes irritates me.
Since PyO3 aims to imitate the behavior of Python’s GIL, I know we cannot avoid some kind of unsafety.
But I’m still investigating a better design with limited my resources.

This is all I can say :slight_smile:
Thank you @rkarp @zeroexcuses @zicklag @amsesk for your positive reaction to PyO3.

4 Likes

Two years ago was when pyo3 had just been forked (after PRs to rust-cpython were not responded to). Since then a lot has happened. pyo3 has received (e.g. the datetime module, the guide and pypy support were added), while cpython has only gotten a few new features and is barely maintained. The one thing where pyo3 can't compete is that it's stuck on nightly in the foreseeable future.

With RefCells around rust objects (#342) you can't just call it twice, as wasm-bindgen successfully shows. For python objects (as in PyDict, PyAny, etc.) you can indeed get multiple mutable references, which should be changed long term, but it's still hard to shoot yourself in the foot because you can only have method calls which are wrapped ffi calls.

The point I'm trying to make is not that there isn't UB in pyo3 or we should stop trying to eliminate all UB from pyo3, but I also think that you shouldn't dismiss a whole project just like that. (Like there are soundness issues in rustc that haven't been fixed for years and cpython is written entirely in an inherently unsafe language)

I understand that you have strong feelings about unsafe and UB, but the "should have learned his lesson" bothers me. People have different priorities and different reasons to choose rust, especially since unsafe and UB are only one small part of security and correctness. For many the completeness and performance of actix easily outweigh bugs with UB they'll never hit with their application. See e.g. this thread by the creator of flask. If you don't want to use a certain crate no one forces you to, but I find publicly shaming someone for what he works on in his spare time inappropriate and unfair.

While the cffi in general is still very popular, nowadays you can use cbindgen to generate the headers, milksnake (or my own tool, pyo3-pack, which despite the name can also be used with cffi and rust-cpython) for packaging and shippai for error handling.

5 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.