Using syn::parse::Parser trait outside of a proc macro

[osa1]

I have a proc macro that can generate a lot of code on real-world inputs. I'm trying to profile and optimize it to improve cargo check times.

To profile it, I have a separate branch in my git repo where I make these changes to the main branch:

• Remove #[proc_macro] attribute, convert proc macro function to &str -> TokenStream
• Implement a bin in the crate that reads an input file with proc macro contents that I want to use to profile my proc macro, and passes file contents to the proc macro function modified in previous step. This function ignores the returned TokenStream. It's used to profile proc macro code.

The problem is this new function with type &str -> TokenStream is panicking in runtime when converting proc_macro2::TokenStream into proc_macro::TokenStream. The full code is here: lexgen/lib.rs at profiling · osa1/lexgen · GitHub (the token_stream.into() part). The error message is:

thread 'main' panicked at 'procedural macro API is used outside of a procedural macro'

If you switch to commit 42fee9c, then run cargo run --bin lexgen_test -- bench_data it works fine. The main difference between the working version and the current (broken) version is that in 42fee9c I can use syn::parse_str. With the current version, I can't use syn::parse_str as my parser is now stateful (cannot implement Parse trait), so I need to use syn::parse::Parser trait (Parser in syn::parse - Rust, the FnOnce(ParseStream) -> Result implementation), which apparently does not work when called outside of a proc macro.

Any ideas on how to make this work? Thanks.

[bjorn3]

The proc_macro crate only works inside proc macros. proc_macro2 exists for allowing the proc macro api's to be used outside proc macros. You should use and return proc_macro2::TokenStream instead of proc_macro::TokenStream.

[osa1]

Unfortunately I have to use syn::parse::Parser trait (Parser in syn::parse - Rust), which uses proc_macro::TokenStream rather than proc_macro2::TokenStream.

Is there no way (hacky or otherwise) to convert a proc_macro2::TokenStream to a proc_macro::TokenStream?

[osa1]

OK, figured it out. syn::parse::Parser has a parse2 method that takes proc_macro2::TokenStream argument. This difference is not visible in the documentation, both argument types are rendered as TokenStream.