Help with proc macro: unexpected token, expected `]`

help
1

Hi,
I'm writting a proc macro to generate some wrapper code. When I call syn::parse2, it returns an Err saying that unexpected token, expected ]``. I inspected in debugger and found Input::parse already returns Ok(Input).

I read the doc on syn::parse2 and it says:
This function enforces that the input is fully parsed. If there are any unparsed tokens at the end of the stream, an error is returned.

So maybe the error is coming from unparsed tokens. But I don't find unparsed tokens... Anyone can help me :sob:

Playground

extern crate proc_macro;

use proc_macro::TokenStream;
use syn::{
    braced, bracketed, parenthesized,
    parse::{Parse, ParseStream},
    parse_macro_input,
    punctuated::Punctuated,
    token::{Brace, Comma, Semi},
    Attribute, FnArg, Generics, Ident, Meta, Token, Type,
};

fn parse_meta(input: ParseStream) -> syn::Result<bool> {
    let is_wrapped = if input.parse::<Token![#]>().is_ok() {
        let content;
        let _ = bracketed!(content in input);

        let r = content.to_string();

        if r == "rhi" {
            true
        } else {
            panic!("expected #[rhi] attribute, found #[{}]", r);
        }
    } else {
        false
    };

    Ok(is_wrapped)
}

#[derive(Clone)]
struct FunctionArg {
    is_wrapped: bool,
    arg: FnArg,
}

impl Parse for FunctionArg {
    fn parse(input: ParseStream) -> syn::Result<Self> {
        Ok(FunctionArg {
            is_wrapped: parse_meta(input)?,
            arg: FnArg::parse(input)?,
        })
    }
}

// -> #[rhi] Result<RenderPass>
#[derive(Clone)]
struct FunctionReturn {
    is_wrapped: bool,
    ty: Type,
    generics: Generics,
}

impl Parse for FunctionReturn {
    fn parse(input: syn::parse::ParseStream) -> syn::Result<Self> {
        // #[rhi] Result<GraphicsPipeline, Error>

        let is_wrapped = parse_meta(input)?;

        let ty = input.parse()?;
        let generics = input.parse()?;

        Ok(Self {
            is_wrapped,
            ty,
            generics,
        })
    }
}

// fn bind_shader_resource_group(&mut self, index: u32, #[rhi] srg: &ShaderResourceGroup) -> &mut Self;
#[derive(Clone)]
struct FunctionDecl {
    fn_name: Ident,
    args: Vec<FunctionArg>,

    return_ty: FunctionReturn,
}

impl Parse for FunctionDecl {
    fn parse(input: syn::parse::ParseStream) -> syn::Result<Self> {
        // fn bind_shader_resource_group(&mut self, index: u32, #[rhi] srg: &ShaderResourceGroup) -> &mut Self;

        let _: Token![fn] = input.parse()?;
        let fn_name: Ident = input.parse()?;

        let name_str = fn_name.to_string();
        println!("fn_name: {:?}", name_str);

        let args;
        let _ = parenthesized!(args in input);

        let args = args.parse_terminated(FunctionArg::parse, Token![,]).unwrap();
        let args = args.iter().map(|a| a.to_owned()).collect();

        let _ = input.parse::<Token![->]>().unwrap();

        let return_ty: FunctionReturn = input.parse().unwrap();

        Ok(Self {
            fn_name,
            args,
            return_ty,
        })
    }
}

struct Input {
    trait_name: Ident,
    impl_generics: Generics,

    target: Type,
    target_generics: Generics,

    functions: Vec<FunctionDecl>,
}

impl Parse for Input {
    fn parse(input: syn::parse::ParseStream) -> syn::Result<Self> {
        let _: Token![impl] = input.parse()?;
        let trait_name: Ident = input.parse()?;
        let impl_generics: Generics = input.parse()?;
        let _: Token![for] = input.parse()?;

        let target = input.parse()?;
        let target_generics = input.parse()?;

        let content;
        let _ = braced!(content in input);

        let functions = content.parse_terminated(FunctionDecl::parse, Token![;])?;

        let functions: Vec<FunctionDecl> = functions.iter().map(|f| f.to_owned()).collect();

        Ok(Input {
            trait_name,
            impl_generics,
            target,
            target_generics,
            functions,
        })
    }
}

// #[proc_macro]
// pub fn make_answer(input: TokenStream) -> TokenStream {
//     let input = parse_macro_input!(input as Input);
//     todo!()
// }

#[cfg(test)]
mod test {
    use crate::Input;

    #[test]
    fn test_rhi_proc_macro_parse() {
        use quote::quote;

        let input = quote! {
            impl IRhiGraphicsCommandEncoder<RhiDevice> for RhiGraphicsCommandEncoder<'_> {
                fn create_renderpass(&self, desc: &RenderPassDesc<'_>) -> #[rhi] Result<RenderPass>;

                fn set_vertex_buffer(&mut self, index: u32, #[rhi] buffer: &RhiBuffer, offset: u64) -> &mut Self;
            }
        };

        let input: syn::Result<Input> = syn::parse2(input);

        match input {
            Ok(input) => (),
            Err(e) => {
                let msg = e.to_string();
                panic!("{}", msg);
            }
        }
    }
}
2

Thanks for posting a good reproducing example; unfortunately, I still hat to copy this out into a local project after all to get better error messages, as when used in an actual macro, the error message gets attached to a span neatly allowing the compiler to point precisely to the code location where syn failed parsing; so the error looks as follows

error: unexpected token, expected `]`
 --> src/main.rs:7:65
  |
7 |     fn create_renderpass(&self, desc: &RenderPassDesc<'_>) -> #[rhi] Result<RenderPass>;
  |                                                                 ^^^

pointing to the rhi, which should hopefully be useful in debugging the problem. (Indeed it helps, see below…)

Admitted, achieving this on the playground is hard or impossible, as you’ll need multiple crates depending on each other in order to properly use a proc-macro.

Anyway, let’s try to figure out the issue… reading through all your definitions until the #[rhi] is reached points me to fn parse_meta which is supposed to handle that attribute, so the problem must be there. Re-reading your question, I see you already noted

And I think you’re right. The contents of the #[rhi] you only inspect via content.to_string(), however this use of Display on the ParseStream likely does not count as actually “parsing” the contents.

Without being 100% familiar with syn myself, options that I could find include:

  • Parse the buffer/stream in question into a proc_macro2::TokenStream (supported from this Parse impl)[1] as a general solution to fully consume a ParseBuffer.

    let _ = bracketed!(content in input);

let content = content.parse::<proc_macro2::TokenStream>()?;

let r = content.to_string();

if r == "rhi" {
    true
} else {
    panic!("expected #[rhi] attribute, found #[{}]", r);
}

    (playground)

  • for your use-case, define yourself a rhi “custom keyword” and use that (which also replaces the need for the subsequent == "rhi")

    mod kw {
    syn::custom_keyword!(rhi);
}

    let _ = bracketed!(content in input);

let _: kw::rhi = content.parse()?;

true

    (playground)

  1. note that the Display implementation you used before also creates a TokenStream anyway, just without also “consuming” the input; so this isn’t actually all that different ↩︎

1 Like
3

Hi @steffahn,

Thank you for your detailed answer! Your solutions solve my problem perfectly :smile:

4

It is only hard, not impossible. Like this:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=48d0807ad279489a9f1c9d90ab540a2a

8 Likes
5

That’s very neat, I didn’t know about #![crate_type = "proc-macro"] at all – so my intuition of “hard” to achieve was rather based on ideas involving creation of other source files and manually invoking the compiler on them from the playground.

For anyone else wondering about the role of the #[test] “annotation”, that seems to just be a line of documentation without any “actual” function (/*! … */ is an inner doc comment for the surrounding module; so the code block becomes a doc-test anyway), but it triggers the playground UI’s simple heuristic to think “oh, there’s tests present!” and to make the “TEST” button action the default/preselected.

1 Like