PyArray -- make array of compound type

1

Lest this is an XY question: I want to pass a 2-dimensional ndarray::Array2 of a custom structure to Python.

If this doesn't make sense I'll try to write a minimally expressive example -- won't be minimally functional, because my problem is that it won't compile.

This, hopefully, distills down my 600-line file with one little ol' problem down to something intelligible. My problem is that when I try to call from_array on a ndarray of an arbitrary object, I get the error the trait `numpy::Element` is not implemented for `SomethingArbitrary`

So -- is there a way take my ndarray::Array2 of some weird type and convert it into a numpy::PyArray2 of the same type? I'm probably going to end up converting to a vector, and then to a PyArray2, but that seems very messy.

use pyo3::prelude::*;
use pyo3::exceptions::*;
use numpy::{PyArray2};
use ndarray::{Array2};

struct SomethingArbitrary {
  this: u16,
  that: bool,
}
...

let r: Array2::<SomethingArbitrary> = make_array_of_something_arbitrary();
pyo3::Python::with_gill(|py| {
  PyArray::from_array(py, r)
})
2

Well, now, isn't that interesting.

I implemented numpy::Element for SomethingArbitrary:

impl Element for SomethingArbitrary {
    const DATA_TYPE: numpy::DataType = numpy::DataType::Object;

    fn is_same_type(dtype: &numpy::PyArrayDescr) -> bool {
        dtype.get_datatype().unwrap() == numpy::DataType::Object
    }
}

Then just used to_pyarray:

pyo3::Python::with_gil(|py| {
  let result = self.correct(r);
  result.to_pyarray(py).to_object(py)
})

It compiles just fine! Everything's wonderful! Except that when I try to run the containing function from Python I get a seg fault! I'm feeling kinda cheated here, because I didn't call any unsafe code in this chain!

3

So now this is turning into "should I file a bug report?"

Here's a fairly easily reproducible example.

lib.rs:

use numpy::{Element, PyArray2, ToPyArray};
use pyo3::prelude::*;

#[pyclass]
#[derive(Debug, Clone)]
struct SomeStruct {
    #[pyo3(get)]
    pub bob:    bool,
    #[pyo3(get)]
    pub ralph:  u16,
}

impl Element for SomeStruct {
    const DATA_TYPE: numpy::DataType = numpy::DataType::Object;

    fn is_same_type(dtype: &numpy::PyArrayDescr) -> bool {
        dtype.get_datatype().unwrap() == numpy::DataType::Object
    }
}

#[pyfunction]
pub fn gen_some() -> PyObject {
    let bob = Array2::<SomeStruct>::from_shape_fn((3, 4), |(n, m)| {
        SomeStruct {
            bob: (n * m) % 2 == 0,
            ralph: ((n * m) % 65536) as u16,
        }
    });

    pyo3::Python::with_gil(|py| {
        bob.to_pyarray(py).to_object(py)
    })
}

#[pyfunction]
fn gen_one(bob: bool, ralph: u16) -> SomeStruct {
    SomeStruct {bob: bob, ralph: ralph}
}

#[pymodule]
fn pyo3_numpy(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(gen_some, m)?)?;
    m.add_function(wrap_pyfunction!(gen_one, m)?)?;
    m.add_class::<SomeStruct>()?;
    Ok(())
}

Cargo.toml. Note that with the versions mentioned, you need to run cargo update --package ndarray:0.15.3 --precise 0.14.0 to get this to work. But if you go with the latest of everything the problem still happens.

[package]
name = "pyo3_numpy"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
crate-type = ["cdylib"]

[dependencies]
# Need to run `cargo update --package ndarray:0.15.3 --precise 0.14.0
ndarray = "0.14"
numpy   = "0.14"
pyo3    = "0.14"
4

Segfault with no unsafe? Sounds like it to me.

1 Like
5

Sounds like the developers know that you can't do it: Refactor Element trait by kngwyu · Pull Request #143 · PyO3/rust-numpy · GitHub. But it would be nice if they built a fence around it for the naive.

6

If anyone runs into this: a workaround that's almost nice is to just make a vector of vectors of the object (or one vector if it's 1D, or n if it's N-D). Python sees it as a list (of lists, of lists, depending on how many dimensions). You can put that directly into a call tonumpy.array.

So instead of your Python code saying thingie = get_thingie(), it says thingie = numpy.array(get_thingie()). Which isn't too far off, and will still work if, in the future, the problem gets fixed and you change your library code to return an ndarray.

1 Like
7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.