How do I replace a function with a block (using proc_macro/syn)

I'm trying to write a macro (stdlibfn) that turns a function within another function, into a block. For context, this is writing a little macro library to make it easier to create stdlib functions for my language - I have that working, now I want to automatically add them to the stdlib. For example, I'd write:

fn stdlib() -> Stdlib {
  let fns : Vec<String, MyFunctionDeclaration> = vec![];
  #[stdlibfn]
  fn int__random__0() {
    dint(rand::random());
  }
  fns.into_iter().collect()
}

and have that turn into:

fn stdlib() -> Stdlib {
  let fns = vec![];
  {   
    fn int__random__0() -> (String, MyFunctionDeclaration) {
       // This bit works, so omitting
    }
    fns.push_back(int_random_0());
  }
  fns.into_iter().collect()
}

Note that this replaces an ItemFn with an ExprBlock, which syn doesn't seem to like. When I try to do this I get errors like "error: macro expansion ignores token { and any following", which implies to me that it doesn't like the idea of replacing an ItemFn with an ExprBlock? I've tried various other formulations of a similar sort, with no luck.

Here's my code:


#[proc_macro_attribute]
pub fn stdlibfn(_attr: TokenStream,
                item: TokenStream)
                -> TokenStream {
  let input = syn::parse_macro_input!(item as syn::ItemFn);
  // Note extra brace wrapper to make a block, 2 lines below
  let fn_stmt: syn::Stmt = parse_quote! {
    {
      #[allow(non_snake_case)]
      fn #fn_name() -> (String, MyfunctionDeclaration) {
        // This bit works, so omitting
      }
    } 
  };
  println!("{:?}", fn_stmt); // confirmed that this is parsed OK.
  let insert_output: syn::Stmt = parse_quote! {
    #fn_name();
  };
  println!("{:?}", insert_output); // This also parses fine.
  let block = ExprBlock { attrs: vec![],
                          label: None,
                          block:
                            Block { stmts: vec![fn_stmt, insert_output],
                                    brace_token: token::Brace { span: Span::call_site(), }, }, };

  TokenStream::from(block.into_token_stream()) // this causes the error below
  // TokenStream::from(fn_stmt.into_token_stream()) // if I remove the {} and use this, the function creation works fine.

and this is the error:

error: macro expansion ignores token `{` and any following
  --> src/eval.rs:44:3
   |
44 |   #[stdlibfn]
   |   ^^^^^^^^^^^ caused by the macro expansion here
   |
   = note: the usage of `stdlibfn!` is likely invalid in item context

I don't have to return a block, but I want to find a way to turn it into some sort of list of statements so that I can add it to the Vec automatically.

I shouldn't be using an ItemFn, but syn doesn't appear to have another type to represent a function within a function, and an ItemFn parses fine in that context (it only fails when I try to return an ExprBlock).

Is there a way to accomplish this? Thanks!

It sounds like you want a way to create a bunch of functions and have them added to a collection mapping the function's name to a MyFunctionDeclaration (presumably a combination of the function signature and a pointer to the original function).

What I did was first create a MyFunctionDeclaration type that different functions can be converted into (I'm assuming you've already implemented this bit).

pub struct MyFunctionDeclaration {
    // real implementation here
}

impl From<fn()> for MyFunctionDeclaration {
    fn from(_f: fn()) -> MyFunctionDeclaration {
        todo!()
    }
}

impl From<fn(&str) -> u32> for MyFunctionDeclaration {
    fn from(_f: fn(&str) -> u32) -> MyFunctionDeclaration {
        todo!()
    }
}

Then wrote a macro_rules macro which matches zero-or-more functions and adds them to a HashMap<String, MyFunctionDeclaration> (my stand-in for Stdlib).

use std::collections::HashMap;

pub type Stdlib = HashMap<String, MyFunctionDeclaration>;

macro_rules! function_map {
    ( $(
        // match a function signature and body
        fn $name:ident( $($arg_name:ident : $arg_ty:ty),* $(,)? ) $(-> $ret:ty)?
            $body:block
    )*) => {{
        let mut functions: Stdlib = HashMap::new();

        $(
            // declare the function
            fn $name( $($arg_name : $arg_ty)*) $(-> $ret)? $body
            // coerce it to a function pointer
            let function_pointer: fn( $($arg_ty),*) $(-> $ret)? = $name;
            // and add it to our hashmap, converting to a MyFunctionDeclaration
            functions.insert(stringify!($name).to_string(), function_pointer.into());
        )*

        functions
    }}
}

And the final use site looks like this:

pub fn stdlib() -> Stdlib {
    function_map! {
        fn int_random_0() {
            todo!()
        }

        fn int_random_2(_some_string: &str) -> u32 {
            todo!()
        }
    }
}

(playground)

This is a great idea, thanks!

It looks like @Michael-F-Bryan has XY-ed the OP, so such post may not be that interesting anymore.
Still, it may be useful for people out there to understand what was going on:


Regarding the OP, I don't think you can achieve what you wanted with a procedural macro attribute.

At least not on one that takes a function directly. Rather, you'd need to apply the attribute on a braced expression, containing the function definition, as follows:

fn stdlib () -> Stdlib {
    let fns ...;
    #[stdlibfn] {
        fn int__random__0 ()
        {
            dint(::rand::random());
        }
    }
}

Which cannot currently be done on stable Rust, since attributes cannot be applied to expressions directly.


Why this odd requirement? The thing is, one has to understand that the Rust parser can, at "any" point, parse an expression, a statement, or an item, such as a type or function definition.

An expression can be of the block kind, in which case it can contain several statements, such as a terminating expression statement. Another kind of statement is the item definition statement, which is when Rust will expect an item definition inside it.

Example

The code { fn foo () {} }, when parsed as an Expr, yields:

Expr::Block(
    ExprBlock {
        attrs: [],
        label: None,
        block: Block {
            brace_token: Brace,
            stmts: [
                Stmt::Item(
                    Item::Fn(
                        ItemFn {
                            attrs: [],
                            vis: Inherited,
                            sig: Signature {
                                constness: None,
                                asyncness: None,
                                unsafety: None,
                                abi: None,
                                fn_token: Fn,
                                ident: Ident {
                                    sym: foo,
                                    span: bytes(6..9),
                                },
                                generics: Generics {
                                    lt_token: None,
                                    params: [],
                                    gt_token: None,
                                    where_clause: None,
                                },
                                paren_token: Paren,
                                inputs: [],
                                variadic: None,
                                output: Default,
                            },
                            block: Block {
                                brace_token: Brace,
                                stmts: [],
                            },
                        },
                    ),
                ),
            ],
        },
    },
)

Which, to "summarize", could be written as:

this -> // {
is      //      fn foo () {} ; // <- this is a statement inside the block expr (semicolon can be elided)
an expr // } // ^^^^^^^^^^^^
                this is an item definition

By the time you apply a procedural macro attribute to an item (e.g., when you apply it to the function definition directly; if you apply it to the wrapping braces, you'd be applying the attribute to an expression), Rust will call the procedural macro, as a function, to generate the expanded source code. But such expanded source code is "doomed" to still belong to that item(s)-as-statement(s) category, meaning that you will only be able to expand to a sequence of items. And a ::syn::Expr (the one you tried to generate) is no item. Hence the error.

And once you see that you are writing #[attr] { ... }, which on top of that, only works on nightly, you realize that we have gone a long way to just end up writing something very similar to our good old:

macro! { ... }

So, if you were to try and solve the problem in the OP, you'd simply have to write:

fn stdlib () -> Stdlib {
    let fns ...;
    stdlibfn! { // <- allowed to expand to an expr (as of 1.45.0, or using proc-macro-hack)
        fn int__random__0 ()
        {
            dint(::rand::random());
        }
    }
}
an advanced alternative

Or you could do:

#[stdlibfn]
fn stdlib () -> Stdlib {
    let fns ...;
    #[stdlibfn]
    fn int__random__0 ()
    {
        dint(::rand::random());
    }
}

by using the "preprocessor pattern": the outer macro will take care, instead of the Rust compiler, of handling everything that goes inside the function's body. So you can try and detect a fake inner #[stdlibfn] attribute, and remove both the attribute and the annotated item, by replacing all that with a Block expression, thanks to the outer macro having access to all the statements inside the Block that defines the body of stdlib().

2 Likes

Thanks, appreciate the explanation of how it all works, and the ideas.

My macro is doing a lot more than that, I just omitted it because it's not relevant to the question.

In hindsight that wasn't the most clear usage of an impersonal you. I intended to mean:

And once we see that we are are writing #[attr] { ... } ... we realize ...

I just wanted to follow the flow of ideas:

  1. We can't have #[name] <fn item> expand to an expression or block,

  2. The closest thing to achieve that would be to write #[name] { <fn item> }

  3. At that point, writing name! { <fn item> } looks more natural and achieves the same thing :slightly_smiling_face:

(But as I said, if you do feel like #[name] <fn item> provides the best ergonomics, which, truth be told, is a paramount thing for a macro to be good, then using the outer attribute as a custom pre-processor would be the best, albeit most complex, course of action).