Modify a numpy array with pyo3.numpy

1

Here's my representative code, within a #[pymethods]-tagged impl block. This is getting called from Python.

    fn py_do_something(&self, frame: &PyArray2<i16>) {
        unsafe {
            let bob = frame.as_array_mut();
            self.do_something(bob);
        }
    }

It works, but as_array_mut() is unsafe, and I'm not entirely clear on why.

So I have two questions, either one of which I'd like to know the answers to:

  1. Why is as_array_mut() unsafe? What do I have to do to assure safety here, since the compiler isn't going to?
  2. The signature of the called function is do_something<'a>(&self, mut source: ArrayViewMut2::<'a, i16>), and the intent is to modify the array in place (hence the mutable ArrayView). Is there some other way to get a mutable array view from a PyArray that is safe (I'm just climbing out of the "dreadful beginner" stage of Rust, and I'm definitely still there for pyo3 and pyo3.numpy -- so even an answer that has me firmly smacking my forehead will be welcome).
2

Note that your function takes an immutable reference to the array but that method returns a mutable view into the array. That is because pyo3's types use "interior mutability".

See GIL, mutability and object types - PyO3 user guide and std::cell - Rust

The unsafe part is that it is undefined behavior to create aliasing &mut. This function gives you that mutable reference without checking. As such it would be really bad if you called it twice and created aliasing mutable references.

To be honest I am surprised there is no checked method for this...

1 Like
3

This as_array_mut is safe; are you referring to another?

1 Like
4

@quinedot thank you! I'm in dependency hell on this particular project, and had made a tactical decision to carry on with the kludge of using back-reved versions of several packages so that I wouldn't get into a version conflict with ndarray (long story -- see the link).

I've been strictly looking at the documentation for 0.13.0, because that's what works with the package combination I have. You made me look at the current documentation, and voila! as_array_mut is safe in the latest version.

There's a suggestion in that thread on how to resolve things which I had not taken -- I'll be trying that, and see if it makes the sun break through the clouds for me.

5

This one is not: PyArray in numpy::array - Rust It looks like it was made unsafe in version 0.10.0

I'm not sure if that means it was unsound before then. Generally it is not a good idea to depend on ffi library versions that are that much older.

6

So that's interesting -- you refer to the documentation on pyo3.github.io -- the documentation for as_array_mut on docs.rs says it's unsafe.

Are you looking at pre-release documentation?

7

Oh, that is interesting. It does seem to be out of date :slightly_frowning_face:. I filed an issue for them.

Here's the PR to make it unsafe which was in part to fix this issue illustrating UB. It looks like the contract is, "don't invoke this if you got a reference (or ArrayView) still hanging around", because they offer unchecked ways to do that. Treat it like casting a *mut more or less.

1 Like
8

I think I'm veering off into whining here, or I'm showing my Rust newbieness -- but shouldn't this have been solved by making so that the behaviour changes depending on whether or not he PyArray was mutable? Or maybe whether the elements are?

Or is there something about the process or wrapping up Python objects into Rust structs that you lose the ability to do that?

9

It could be fixed by returning an Option or panicking at runtime ala RefCell. (Or with some sort of Mutex or other lock acquisition.) Apparently pyo3 had no ability to do that, because they weren't keeping track of who borrowed what at runtime (more like UnsafeCell). This PR implies that they can now, I think? No idea of how that impacts the numpy crate, but maybe you could follow up with them.

You can view this particular issue as arising from being too free and loose with interior mutability. The unsafe closed a soundness hole by putting the onus on the user; a more complete solution would be to offer safe (and sound) ways to use the interior mutability by adding runtime checks.

Is there something about Python that makes the run-time tracking difficult? I'm no expert, but apparently there were some challenges around class inheritance, based on reading the PR text.

10

Python is like C in that anyone with a pointer can write to the data. Python is worse than C in that in C, the "natural" way to express a variable is as the variable itself -- in Python any compound variable (i.e. a class) is actually a reference to the actual variable, which Python maintains under the hood.

So in Python, if I create a class instance and hand it to a thread, both my main-line code and the thread can blithely modify it. Or if I say

bob = SomeClass()
ralph = bob
bob.modify_somehow()

then I've modified both bob and ralph.

(Edit -- I've modified the one thing in the background that bob and ralph both reference. I'm leaving my original wording in, because it's pretty common to forget that something like ralph = bob, in Python, does not mean that "ralph is now a copy of bob" -- it means "ralph is now an alias for bob". AFAIK Rust does pretty much the same thing, except when you say let ralph = bob you retire bob, and can't use it any more.)

I don't think you could (or would want to) bring the level of Rust safety to Python, but bringing that level of safety to whatever happens within the Rust code would be nice.

11

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.