Pyo3: Access functions created through macro are not visible

I'm creating access functions for the Python layer through a macro. Unfortunately the functions are not recognized by the PyClassImplCollector:

#[macro_use]
extern crate paste;

use std::str;
use std::collections::HashMap;

use pyo3::prelude::*;

#[macro_export]
macro_rules! read_access_methods {
    ($function:ident, $($dict:ident).+, $class:ident) => {
        paste! {
            fn [<$function _cnt>](slf: PyRef<Self>) -> PyResult<usize> {
                Ok(slf.data.$($dict).+.len())
            }
        }

        fn $function(slf: PyRef<Self>, key: String) -> PyResult<PyObject> {
            let gil = Python::acquire_gil();
            let py = gil.python();
            match slf.data.$($dict).+.get(key.as_bytes()) {
                Some(v) => Ok($class(v).to_object(py)),
                None => return Ok(py.None())
            }
        }

        paste! {
            fn [<$function s>](slf: PyRef<Self>, pattern: Option<String>) -> PyResult<HashMap<String, PyObject>> {
                let gil = Python::acquire_gil();
                let py = gil.python();
                let mut result: HashMap<String, PyObject> = HashMap::new();
                for (key, value) in &slf.data.$($dict).+ {
                    result.insert(
                        str::from_utf8(&key).unwrap().to_string(),
                        $class(value).to_object(py));
                }
                return Ok(result);
            }
        }
    };
}

#[derive(Debug)]
pub struct Foo {
    pub bar: bool,
}

impl Foo {
    pub fn new() -> Foo {
        Foo { bar: false }
    }
}

pub struct PyFoo<'a>(pub &'a Foo);

impl<'a> ToPyObject for PyFoo<'a>
{
    fn to_object(&self, py: Python) -> PyObject {
        let slf = self.0;
        let mut result: HashMap<String, PyObject> = HashMap::new();
        result.insert("bar".to_owned(), slf.bar.into_py(py));
        result.into_py(py)
    }
}

#[derive(Debug)]
pub struct Data {
    pub foo: HashMap<Vec<u8>, Foo>,
}

impl Data {
    pub fn new() -> Data {
        Data { foo: HashMap::new() }
    }
}

#[pyclass]
#[derive(Debug)]
pub struct Instance {
    pub data: Data,
}

#[pymethods]
impl Instance {
    #[new]
    fn __new__() -> PyResult<Self> {
        Ok(Instance {
            data: Data::new(),
        })
    }

    read_access_methods!(bar, foo, PyFoo);
}

#[pymodule]
#[pyo3(name="foolib")]
fn init(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add("__version__", env!("CARGO_PKG_VERSION"))?;
    m.add_class::<Instance>()?;
    Ok(())
}

With the following command I'm expanding the output:

cargo rustc --profile=check -- -Zunpretty=expanded

As you can see indeed the functions are created as expected, BUT the PyClassImplCollector ignores these functions:

impl Instance {
    fn __new__() -> PyResult<Self> { Ok(Instance{data: Data::new(),}) }
    fn bar_cnt(slf: PyRef<Self>) -> PyResult<usize> { Ok(slf.data.foo.len()) }
    fn bar(slf: PyRef<Self>, key: String) -> PyResult<PyObject> {
        let gil = Python::acquire_gil();
        let py = gil.python();
        match slf.data.foo.get(key.as_bytes()) {
            Some(v) => Ok(PyFoo(v).to_object(py)),
            None => return Ok(py.None()),
        }
    }
    fn bars(slf: PyRef<Self>, pattern: Option<String>)
     -> PyResult<HashMap<String, PyObject>> {
        let gil = Python::acquire_gil();
        let py = gil.python();
        let mut result: HashMap<String, PyObject> = HashMap::new();
        for (key, value) in &slf.data.foo {
            result.insert(str::from_utf8(&key).unwrap().to_string(),


                              PyFoo(value).to_object(py));
        }
        return Ok(result);
    }
}
impl pyo3::class::impl_::PyClassNewImpl<Instance> for
 pyo3::class::impl_::PyClassImplCollector<Instance> {
    fn new_impl(self) -> Option<pyo3::ffi::newfunc> {
        Some({
                 unsafe extern "C" fn __wrap(subtype:
                                                 *mut pyo3::ffi::PyTypeObject,
                                             _args: *mut pyo3::ffi::PyObject,
                                             _kwargs:
                                                 *mut pyo3::ffi::PyObject)
                  -> *mut pyo3::ffi::PyObject {
                     use pyo3::callback::IntoPyCallbackOutput;
                     pyo3::callback::handle_panic(|_py|
                                                      {
                                                          let _args =
                                                              _py.from_borrowed_ptr::<pyo3::types::PyTuple>(_args);
                                                          let _kwargs:
                                                                  Option<&pyo3::types::PyDict> =
                                                              _py.from_borrowed_ptr_or_opt(_kwargs);
                                                          let result =
                                                              Instance::__new__();
                                                          let initializer:
                                                                  pyo3::PyClassInitializer<Instance> =
                                                              result.convert(_py)?;
                                                          let cell =
                                                              initializer.create_cell_from_subtype(_py,
                                                                                                   subtype)?;
                                                          Ok(cell as
                                                                 *mut pyo3::ffi::PyObject)
                                                      })
                 }
                 __wrap
             })
    }
}

Any help?

Can you try using that macro to create an entire new #[pymethods] block rather than being in one?

To do that you'd also have to use PyO3's multiple-pymethods feature.

As for why this doesn't work, I suspect that the pymethods macro runs before your macro is evaluated. It would have to be evaluated the other way around. I don't know if there's a way to do that.

How can I enable this feature?

Just like how you can use any other crate's features. For pyo3's features see pyo3 - Rust . For using features in general, see Features - The Cargo Book

I've found out how to enable this feature by looking into the sources to pyo3.

#[cfg(feature = "multiple-pymethods")]

But it seem it doesn't help solving this issue.

To clarify, I think you need to use multiple #[pymethods] blocks here - one for your __new__ function, one that your macro generates, rather than trying to nest them in one block. PyO3 doesn't support this out of the box, which is why you need to use that feature.

(also, if you use a proc macro rather than a macro_rules, you can annotate your structs with it)

But it seem it doesn't help solving this issue.

That's not how you invoke a crate's feature, you need to turn this on in your Cargo.toml:

[dependencies.pyo3]
version = "0.14.1"
features = ["multiple-pymethods"]

I think your macro needs to expand to the whole thing:

#[macro_export]
macro_rules! read_access_methods {
    ($instance:ident, $function:ident, $($dict:ident).+, $class:ident) => {
#[pymethods]
impl $instance {
    #[new]
    fn __new__() -> PyResult<Self> {
        Ok($instance {
            data: Data::new(),
        })
    }
     /* rest of macro */
    }
}

That way your macro is evaluated first and the expanded output is fed into pymethods.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.