Unable to parse a TokenStream into a ParseBuffer with the syn crate

help
1

I'm using the syn crate and running into an issue when attempting to override the visit_macro_mut function in order to visit and mutate inputs to macros (in a non-procedural macro context). I tried using the parse_macro_input! macro at first, but the documentation for it seems to imply that that macro must be called in a function that receives and returns a TokenStream , which I'm not doing. So I have something like this:

fn visit_macro_mut(&mut self, node: &mut Macro) {
  let tokens = node.tokens.clone();
  let tokens = tokens.into();
  let parsed = Punctuated::<Expr, Token![,]>::parse_terminated(tokens).unwrap();
}

However the compiler complains about the tokens.into() call, which I'm using to attempt to transform tokens: TokenStream into tokens: ParseBuffer. It reports back the following:

the trait bound `&syn::parse::ParseBuffer<'_>:
std::convert::From<proc_macro2::TokenStream>` is not satisfied

`std::convert::From<proc_macro2::TokenStream>` is implemented for `&mut syn::parse::ParseBuffer<'_>`, but not for `&syn::parse::ParseBuffer<'_>`

So I'm not sure how to coerce tokens in order to get this to work, or if there's a different way in which I should be going about this.

2

Looking just at the error, I can recommend the following change:

let parsed = Punctuated::<Expr, Token![,]>::parse_terminated(&tokens).unwrap();

Note the &. This way we tell that the reference to tokens will be the thing parse_terminated wants, and not the tokens itself, so tokens will be inferred to be the ParseBuffer.

Another way to debug such errors is to add types explicitly:

let tokens: ParseBuffer = tokens.into();
let parsed = Punctuated::<Expr, Token![,]>::parse_terminated(tokens).unwrap();

Playground
In this case, you'll see two errors:

error[E0308]: mismatched types
 --> src/lib.rs:9:64
  |
9 |   let parsed = Punctuated::<Expr, Token![,]>::parse_terminated(tokens).unwrap();
  |                                                                ^^^^^^
  |                                                                |
  |                                                                expected `&syn::parse::ParseBuffer<'_>`, found struct `syn::parse::ParseBuffer`
  |                                                                help: consider borrowing here: `&tokens`

error[E0277]: the trait bound `syn::parse::ParseBuffer<'_>: std::convert::From<proc_macro2::TokenStream>` is not satisfied
 --> src/lib.rs:8:36
  |
8 |   let tokens: ParseBuffer = tokens.into();
  |                                    ^^^^ the trait `std::convert::From<proc_macro2::TokenStream>` is not implemented for `syn::parse::ParseBuffer<'_>`
  |
  = note: required because of the requirements on the impl of `std::convert::Into<syn::parse::ParseBuffer<'_>>` for `proc_macro2::TokenStream`

The first error is roughly the same I've said before: you're converting value to ParseBuffer, but parse_terminated expects &ParseBuffer.

The second error, however, is more important: it says that you can't convert TokenStream to ParseBuffer either. And this, in fact, is understandable, since syn docs explicitly say:

There is no public way to construct a ParseBuffer . Instead, if you are looking to invoke a parser function that requires ParseStream as input, you will need to go through one of the public parsing entry points.

3

I also got a response from dtolnay in a separate channel. He recommended using parse_body_with and passing in the parser: node.parse_body_with(Punctuated::<Expr, Token![,]>::parse_terminated). That resolved the issue with my being unable to coerce a TokenStream into a ParseStream.

So you were right in your analysis that the second error was more important. I needed to use a syn::parse* function.

Thanks!

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.