How to write a `proc_macro2::TokenStream` back to a file and let it could be compiled as a normal source file by rustc?

help
1

Hi guys,
I'm trying to implement a general codegen feature with syn, quote and proc_macro2. My idea is that using syn::parse_file to get AST of a file, running some analysis and codegen steps, then quote! my generated AST structures into proc_macro2::TokenStream, and write generated code back to a file with proc_macro2::TokenStream::to_string().

Unfortunately the ToString implementation of TokenStream does not behave as what I expected(and maybe I should have discovered it before I start writing my codegen feature...). That is, the ToString leave a space between every two tokens, and any whitespaces of original input file is discarded: that is what a lexer should do. Like following:

source code file:

fn main() {
    println!("Hello, world!");
}

output string:

fn main () { println ! ("Hello, world!") ; }

But obviously such output string could not be compiled by Rustc because it's not a normal rust source code file anymore. For whitespaces in the input file, if the output is still a rust source code file we could simply run rustfmt on it, however it's not. And I didn't find any other methods to convert TokenStream back to a string(and so as a file).

Is there any possible solution for my general purpose codegen feature? For my work this is necessary and could not be replaced with proc_macros, because I must generate new files from some source files, which can't be achieved by proc_macros.

2

This is valid Rust source code. You can run it on the Playground.

There are reasons that a macro expansion might not be a valid source file, but they have to do with spans and identifier hygiene, not whitespace, and they can be avoided when you control the macros that are in use.

2 Likes
3

Thanks, this seems a little anti-intuitive, but actually very reasonable, otherwise the lexer of rustc itself cannot turn source code files into tokens. And I found a lightweight crate which can be used to format generated codes fast: GitHub - dtolnay/prettyplease: A minimal `syn` syntax tree pretty-printer

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.