How do I replace a function with a block (using proc_macro/syn)

[pbiggar]

I'm trying to write a macro (stdlibfn) that turns a function within another function, into a block. For context, this is writing a little macro library to make it easier to create stdlib functions for my language - I have that working, now I want to automatically add them to the stdlib. For example, I'd write:

fn stdlib() -> Stdlib {
  let fns : Vec<String, MyFunctionDeclaration> = vec![];
  #[stdlibfn]
  fn int__random__0() {
    dint(rand::random());
  }
  fns.into_iter().collect()
}

and have that turn into:

fn stdlib() -> Stdlib {
  let fns = vec![];
  {   
    fn int__random__0() -> (String, MyFunctionDeclaration) {
       // This bit works, so omitting
    }
    fns.push_back(int_random_0());
  }
  fns.into_iter().collect()
}

Note that this replaces an ItemFn with an ExprBlock, which syn doesn't seem to like. When I try to do this I get errors like "error: macro expansion ignores token { and any following", which implies to me that it doesn't like the idea of replacing an ItemFn with an ExprBlock? I've tried various other formulations of a similar sort, with no luck.

Here's my code:

#[proc_macro_attribute]
pub fn stdlibfn(_attr: TokenStream,
                item: TokenStream)
                -> TokenStream {
  let input = syn::parse_macro_input!(item as syn::ItemFn);
  // Note extra brace wrapper to make a block, 2 lines below
  let fn_stmt: syn::Stmt = parse_quote! {
    {
      #[allow(non_snake_case)]
      fn #fn_name() -> (String, MyfunctionDeclaration) {
        // This bit works, so omitting
      }
    } 
  };
  println!("{:?}", fn_stmt); // confirmed that this is parsed OK.
  let insert_output: syn::Stmt = parse_quote! {
    #fn_name();
  };
  println!("{:?}", insert_output); // This also parses fine.
  let block = ExprBlock { attrs: vec![],
                          label: None,
                          block:
                            Block { stmts: vec![fn_stmt, insert_output],
                                    brace_token: token::Brace { span: Span::call_site(), }, }, };

  TokenStream::from(block.into_token_stream()) // this causes the error below
  // TokenStream::from(fn_stmt.into_token_stream()) // if I remove the {} and use this, the function creation works fine.

and this is the error:

error: macro expansion ignores token `{` and any following
  --> src/eval.rs:44:3
   |
44 |   #[stdlibfn]
   |   ^^^^^^^^^^^ caused by the macro expansion here
   |
   = note: the usage of `stdlibfn!` is likely invalid in item context

I don't have to return a block, but I want to find a way to turn it into some sort of list of statements so that I can add it to the Vec automatically.

I shouldn't be using an ItemFn, but syn doesn't appear to have another type to represent a function within a function, and an ItemFn parses fine in that context (it only fails when I try to return an ExprBlock).

Is there a way to accomplish this? Thanks!

[Michael-F-Bryan]

It sounds like you want a way to create a bunch of functions and have them added to a collection mapping the function's name to a MyFunctionDeclaration (presumably a combination of the function signature and a pointer to the original function).

What I did was first create a MyFunctionDeclaration type that different functions can be converted into (I'm assuming you've already implemented this bit).

pub struct MyFunctionDeclaration {
    // real implementation here
}

impl From<fn()> for MyFunctionDeclaration {
    fn from(_f: fn()) -> MyFunctionDeclaration {
        todo!()
    }
}

impl From<fn(&str) -> u32> for MyFunctionDeclaration {
    fn from(_f: fn(&str) -> u32) -> MyFunctionDeclaration {
        todo!()
    }
}

Then wrote a macro_rules macro which matches zero-or-more functions and adds them to a HashMap<String, MyFunctionDeclaration> (my stand-in for Stdlib).

use std::collections::HashMap;

pub type Stdlib = HashMap<String, MyFunctionDeclaration>;

macro_rules! function_map {
    ( $(
        // match a function signature and body
        fn $name:ident( $($arg_name:ident : $arg_ty:ty),* $(,)? ) $(-> $ret:ty)?
            $body:block
    )*) => {{
        let mut functions: Stdlib = HashMap::new();

        $(
            // declare the function
            fn $name( $($arg_name : $arg_ty)*) $(-> $ret)? $body
            // coerce it to a function pointer
            let function_pointer: fn( $($arg_ty),*) $(-> $ret)? = $name;
            // and add it to our hashmap, converting to a MyFunctionDeclaration
            functions.insert(stringify!($name).to_string(), function_pointer.into());
        )*

        functions
    }}
}

And the final use site looks like this:

pub fn stdlib() -> Stdlib {
    function_map! {
        fn int_random_0() {
            todo!()
        }

        fn int_random_2(_some_string: &str) -> u32 {
            todo!()
        }
    }
}

(playground)

[pbiggar]

This is a great idea, thanks!

[Yandros]

It looks like @Michael-F-Bryan has XY-ed the OP, so such post may not be that interesting anymore.
Still, it may be useful for people out there to understand what was going on:

---

Regarding the OP, I don't think you can achieve what you wanted with a procedural macro attribute.

At least not on one that takes a function directly. Rather, you'd need to apply the attribute on a braced expression, containing the function definition, as follows:

fn stdlib () -> Stdlib {
    let fns ...;
    #[stdlibfn] {
        fn int__random__0 ()
        {
            dint(::rand::random());
        }
    }
}

Which cannot currently be done on stable Rust, since attributes cannot be applied to expressions directly.

---

Why this odd requirement? The thing is, one has to understand that the Rust parser can, at "any" point, parse an expression, a statement, or an item, such as a type or function definition.

An expression can be of the block kind, in which case it can contain several statements, such as a terminating expression statement. Another kind of statement is the item definition statement, which is when Rust will expect an item definition inside it.

Example

The code { fn foo () {} }, when parsed as an Expr, yields:

Expr::Block(
    ExprBlock {
        attrs: [],
        label: None,
        block: Block {
            brace_token: Brace,
            stmts: [
                Stmt::Item(
                    Item::Fn(
                        ItemFn {
                            attrs: [],
                            vis: Inherited,
                            sig: Signature {
                                constness: None,
                                asyncness: None,
                                unsafety: None,
                                abi: None,
                                fn_token: Fn,
                                ident: Ident {
                                    sym: foo,
                                    span: bytes(6..9),
                                },
                                generics: Generics {
                                    lt_token: None,
                                    params: [],
                                    gt_token: None,
                                    where_clause: None,
                                },
                                paren_token: Paren,
                                inputs: [],
                                variadic: None,
                                output: Default,
                            },
                            block: Block {
                                brace_token: Brace,
                                stmts: [],
                            },
                        },
                    ),
                ),
            ],
        },
    },
)

• Playground

Which, to "summarize", could be written as:

this -> // {
is      //      fn foo () {} ; // <- this is a statement inside the block expr (semicolon can be elided)
an expr // } // ^^^^^^^^^^^^
                this is an item definition

By the time you apply a procedural macro attribute to an item (e.g., when you apply it to the function definition directly; if you apply it to the wrapping braces, you'd be applying the attribute to an expression), Rust will call the procedural macro, as a function, to generate the expanded source code. But such expanded source code is "doomed" to still belong to that item(s)-as-statement(s) category, meaning that you will only be able to expand to a sequence of items. And a ::syn::Expr (the one you tried to generate) is no item. Hence the error.

And once you see that you are writing #[attr] { ... }, which on top of that, only works on nightly, you realize that we have gone a long way to just end up writing something very similar to our good old:

macro! { ... }

So, if you were to try and solve the problem in the OP, you'd simply have to write:

fn stdlib () -> Stdlib {
    let fns ...;
    stdlibfn! { // <- allowed to expand to an expr (as of 1.45.0, or using proc-macro-hack)
        fn int__random__0 ()
        {
            dint(::rand::random());
        }
    }
}

an advanced alternative

Or you could do:

#[stdlibfn]
fn stdlib () -> Stdlib {
    let fns ...;
    #[stdlibfn]
    fn int__random__0 ()
    {
        dint(::rand::random());
    }
}

---

by using the "preprocessor pattern": the outer macro will take care, instead of the Rust compiler, of handling everything that goes inside the function's body. So you can try and detect a fake inner #[stdlibfn] attribute, and remove both the attribute and the annotated item, by replacing all that with a Block expression, thanks to the outer macro having access to all the statements inside the Block that defines the body of stdlib().

[pbiggar]

Thanks, appreciate the explanation of how it all works, and the ideas.

@Yandros

And once you see that you are writing #[attr] { ... } , which on top of that, only works on nightly , you realize that we have gone a long way to just end up writing something very similar to our good old:

My macro is doing a lot more than that, I just omitted it because it's not relevant to the question.

[Yandros]

In hindsight that wasn't the most clear usage of an impersonal you. I intended to mean:

And once we see that we are are writing #[attr] { ... } ... we realize ...

I just wanted to follow the flow of ideas:

1. We can't have #[name] <fn item> expand to an expression or block,

2. The closest thing to achieve that would be to write #[name] { <fn item> }

3. At that point, writing name! { <fn item> } looks more natural and achieves the same thing

(But as I said, if you do feel like #[name] <fn item> provides the best ergonomics, which, truth be told, is a paramount thing for a macro to be good, then using the outer attribute as a custom pre-processor would be the best, albeit most complex, course of action).