Can't pass variable as reference - How to keep efficiency? PyO3

I have a vector data which will hold thousand of data-items. I need to pass this vector to a function executing PyO3. The following code works if I pass data is a value, but when I pass it as reference the code won't compile because PyO3 needs ownership of the variable.

use pyo3::types::PyAny;
use pyo3::prelude::*;

#[pyclass]
pub struct SomeItem {
    #[pyo3(get, set)]
    pub a: u64,
    #[pyo3(get, set)]
    pub b: String,
}


fn main() {
    let mut data = std::vec::Vec::<SomeItem>::new();

    for i in 0..10 {
        data.push(SomeItem {
            a: i,
            b: format!("str-{}", i),
        });
    }

    execute_python(&data);
}

// data can't be passed as reference.
fn execute_python(data: &std::vec::Vec::<SomeItem>) {
    pyo3::prepare_freethreaded_python();
    let python_code = std::fs::read_to_string("test.py").expect("Failed to read file...");
    Python::with_gil(|py| {
        let fun: Py<PyAny> = PyModule::from_code(
            py,
            &*python_code,
            "",
            "",
        ).unwrap()
        .getattr("func").unwrap().into();

        fun.call1(py, (data,));
    });
}

I am worried that passing data as value (it will be passed multiple times/minute) will cause a huge inefficiency. What would be a good solution?

Current solution:

use pyo3::types::PyAny;
use pyo3::types::PyList;
use pyo3::prelude::*;

#[pyclass]
pub struct SomeItem {
    #[pyo3(get, set)]
    pub a: u64,
    #[pyo3(get, set)]
    pub b: String,
}


fn main() {
    Python::with_gil(|py| {
        let mut data = std::vec::Vec::<Py<SomeItem>>::new();

        for i in 0..10 {
            let item = SomeItem {
                a: i,
                b: format!("str-{}", i),
            };

            data.push(Py::new(py, item).unwrap());
        }

        execute_python(&data);
        
    });
}

// data can't be passed as reference.
fn execute_python(data: &std::vec::Vec::<Py<SomeItem>>) {
    pyo3::prepare_freethreaded_python();
    let python_code = std::fs::read_to_string("test.py").expect("Failed to read file...");
    Python::with_gil(|py| {
        let py_list = PyList::new(py, data);
        let fun: Py<PyAny> = PyModule::from_code(
            py,
            &*python_code,
            "",
            "",
        ).unwrap()
        .getattr("func").unwrap().into();

        fun.call1(py, (py_list,));
    });
}

Wondering if let py_list = PyList::new(py, data); is efficient.

Fundamentally if the data is going to show up as a Python list on the Python side of the bridge, copying is necessary. The only way to avoid this is to take the numpy approach and expose an object which acts similar to a list on the Python side but isn't a list. PyO3 does not appear to provide such functionality out of the box.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.