How to extend impl block with a proc macro?

I've got a struct with an existing implementation:

struct Foo {
  a: String,
  b: String,
  c: u64,
}

impl Foo {
  pub fn get_a(&self) -> String {
    self.a.clone()
  }

  pub fn set_a(&mut self, a: String) {
    self.a = a;
  }
}

Now I want to extend the existing impl block through a macro:


#[access_methods]
#[methods = ((b, String), (c, u64))]
impl Foo {
  ...
}

The macro should create the getter and setter methods including the corresponding type. supplied by the methods attribute.
What I've found so far is a working example to extend the struct itself with additional fields by a proc macro, but how can I extend an existing impl block?

Sounds like you want an attribute macro.

Alternately, you could have a regular function macro that just expands to an impl block (you can have as many as you like). Or, if you want to be fancy, a derive macro that uses a custom attribute to define the accessors directly on the fields.

1 Like

I've tried it with a declarative macro, but compilation fails with:
error: macros cannot be used as items in #[pymethods] impl blocks

What I didn't mention is that the impl block already has an attribute macro from pyo3: pymethods.
Declarative macros and attribute macros seem to not play well together.
I also think that an attribute macro is the way to go, but I don't know how to implement it.

Okay, well, is the reason you want to "extend" the impl that you want pyo3 to see these new methods and do its own transformations on them? You might, might be able to use an attribute macro, so long as you place it before the pyo3 macro, so that when the pyo3 macro runs, it sees the output of your macro. I say "might" because I've never directly tried this myself, but I believe that's how it works.

Off the top of my head, neither do I. If I was doing this, I'd search for the attribute macro entry point (which I linked to you above), then proceed like with any other procedural macro. From the sounds of it, you've already at least somewhat familiar with procedural macros, and an attribute macro is just a different syntax for doing the same thing.

In terms of implementation, you'd want to use syn to parse the annotated item into an impl block, as well as parse the attribute arguments into a custom structure (by implementing the Parse trait on it). Then, insert the extra methods into the parsed impl block (they look simple enough that quote should suffice for constructing them), and spit it back out as the result of the macro.

1 Like

After reading lot's of howtos and lot's of trial and error I've found a working solution. The attribute proc macro is as follows:

#[proc_macro_attribute]
pub fn access_methods(attr: TokenStream, input: TokenStream) -> TokenStream {
    let mut ast = parse_macro_input!(input as syn::ItemImpl);
    let args = parse_macro_input!(attr as syn::ExprTuple);
    for elem in args.elems.iter() {
        if let syn::Expr::Tuple(ele) = elem {
            let mut elems_iter = ele.elems.iter();
            let key = match elems_iter.next() {
                Some(syn::Expr::Path(p)) => Some(
                    &p.path
                        .segments
                        .first()
                        .expect("Failed to extract key")
                        .ident,
                ),
                _ => None,
            }
            .expect("Failed to extract key");
            let objs = match elems_iter.next() {
                Some(syn::Expr::Path(p)) => Some(p.to_token_stream()),
                Some(syn::Expr::Field(p)) => Some(p.to_token_stream()),
                _ => None,
            }
            .expect("Failed to extract objs");
            let class = match elems_iter.next() {
                Some(syn::Expr::Path(p)) => Some(
                    &p.path
                        .segments
                        .first()
                        .expect("Failed to extract class")
                        .ident,
                ),
                _ => None,
            }
            .expect("Failed to extract class");
            // Return item count
            let fn_cnt = quote::format_ident!("{key}_cnt");
            ast.items.push(
                syn::parse::<syn::ImplItem>(
                    quote!(
                        #[doc = " Return number of items"]
                        fn #fn_cnt(slf: PyRef<Self>) -> PyResult<usize> {
                            Ok(slf.data.#objs.len())
                        }
                    )
                    .into(),
                )
                .expect("Failed to parse code"),
            );
            // Return single item
            ast.items.push(
                syn::parse::<syn::ImplItem>(
                    quote!(
                        #[doc = " Return a single item"]
                        #[doc = ""]
                        #[doc = " # Arguments"]
                        #[doc = ""]
                        #[doc = " * `key` - Item name"]
                        pub fn #key(slf: PyRef<Self>, key: String) -> PyResult<PyObject> {
                            Python::with_gil(|py| -> PyResult<PyObject> {
                                match slf.data.#objs.get(key.as_bytes()) {
                                    Some(v) => Ok(#class(v.clone()).to_object(py)),
                                    None => return Ok(py.None())
                                }
                            })
                        }                       )
                    .into(),
                )
                .expect("Failed to parse code"),
            );
            // Return several items
            let fns = quote::format_ident!("{key}s");
            ast.items.push(
                syn::parse::<syn::ImplItem>(
                    quote!(
                        #[doc = " Return several items"]
                        #[doc = ""]
                        #[doc = " # Arguments"]
                        #[doc = ""]
                        #[doc = " * `names` - Optional list of item names"]
                        #[doc = " * `pattern` - Optional glob-style pattern"]
                        #[doc = " * `ignore_case` - Optional ignore case in pattern"]
                        pub fn #fns(
                            slf: PyRef<Self>,
                            names: Option<Vec<String>>,
                            pattern: Option<String>,
                            ignore_case: Option<bool>
                        ) -> PyResult<HashMap<String, PyObject>> {
                            read_filtered_items!(slf, names, pattern, ignore_case, #objs, #class)
                        }                        )
                    .into(),
                )
                .expect("Failed to parse code"),
            );
            // Item setter
            let set_fn = quote::format_ident!("set_{key}");
            ast.items.push(
                syn::parse::<syn::ImplItem>(
                    quote!(
                    fn #set_fn(mut slf: PyRefMut<Self>, key: String, obj: Option<#class>) -> PyObject {
                        Python::with_gil(|py| -> PyObject {
                            if let Some(obj) = obj {
                                // Insert/Replace object
                                match slf.data.#objs.insert(key.as_bytes().to_vec(), obj.0) {
                                    Some(v) => #class(v.clone()).to_object(py),
                                    None => py.None()
                                }
                            } else {
                                // Remove object if it exists
                                match slf.data.#objs.remove(key.as_bytes()) {
                                    Some(v) => #class(v.clone()).to_object(py),
                                    None => py.None()
                                }
                            }
                        })
                    }
                        )
                    .into(),
                )
                .expect("Failed to parse code"),
            );
        }
    }
    ast.into_token_stream().into()
}

Usage is like this:

#[pyclass]
#[derive(Debug)]
pub struct Instance {
    pub path: PathBuf,
    pub loaded: bool,
    pub data: Data,
    pub errors: Vec<ReaderError>,
    pub dt: usize,
}

#[access_methods((
(system_constant, mod_par.system_constants, PyValueType),
(unit, units, PyUnit),
(group, groups, PyGroup)))]
#[pymethods]
impl Instance {
...
}

Struct Data in struct Instance has several members of type HashMap and some members of type struct, which themselves have members of type HashMap. The macro supports these nested structs.

What I still do not understand is why several calls to this macro always use the original version of the struct which results in dropping the results of the previous macro calls.
Thus the following usage would result in creating access functions just for groups:

#[access_methods(((system_constant, mod_par.system_constants, PyValueType),))]
#[access_methods(((unit, units, PyUnit),))]
#[access_methods(((group, groups, PyGroup),))]
#[pymethods]
impl Instance {
...
}

Any ideas why this usage fails?

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.