I have a #[proc_macro] that creates a function and returns it via: TokenStream::from_str(). This returned function is "printed" onto a single line, at the calling site. Is it possible to have it "printed" as a normal multiline function? I see there is the Span and I can kind of see why it is only on a single line. Do I need to manually make the TokenStream with TokenTrees?
I am working in a closed enviroment with Rust 2018 and no access to crates such as "paste" and "quote".
You either have to intentionally include the formatting or run it through something like rustfmt to add the formatting back in. For example cargo-expand can use rustfmt to turn the expanded macro code into something more readable
I did attempt to use '\n' and '\t', but that didn't work, at least when using the "TokenStream::from_str()".
I will look into using "cargo_expand". Thank you
TokenStream's Display impl doesn't add newlines and indentation. This is unaffected by the fact that you added \ns to the raw string from which you parsed it. The token stream doesn't record whitespace between tokens, only the tokens themselves.
I happened to need to format source code in a &str programmatically and this is what I came up with. not sure how resilient it is, but it worked for my needs.
use std::io::{self, Write};
use std::process::{Command, Stdio};
pub fn rustfmt(source: &str) -> io::Result<String> {
let mut process = Command::new("rustfmt")
.arg("--emit=stdout")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()?;
let mut stdin = process.stdin.take().unwrap();
stdin.write_all(source.as_bytes())?;
stdin.flush()?;
drop(stdin);
String::from_utf8(process.wait_with_output()?.stdout)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "Rustfmt output not UTF-8"))
}
Other words, I will have to build the TokenStream by hand then.
@SkiFire13 thank you for the code, but this is doing the same thing I was doing manually and the TokenStream::from_str() will ignore those formatting chars: \n
No, that won't help either. It is exactly the point that the displayed string doesn't have anything to do with how you create the token stream. It will never indent the code. You'll have to format the final string representation itself.
I'm curious what you mean by "printed" here. The token streams generated by proc-macros are consumed by the compiler and it never "sees" the generated code as source code, so whether a TokenStream includes newlines doesn't matter. That's what @H2CO3 is getting at.
However, if you are using the proc-macro2 crate and proc-macros to generate Rust source code in general, you might want to look into the prettyplease crate from dtolnay. It's like rustfmt, except usable as a library on stable and with a tiny impact on build times.
That said, if you are in a closed environment and not allowed to use 3rd party crates you're kinda playing Rust on hard mode... You'll be limited to running your code through rustfmt like @SkiFire13 suggested or writing a formatter yourself.
@H2CO3 & @Michael-F-Bryan are you saying that the Span found within each token, that designates the bytes, isn't related to how the code is "printed" in the code file? Guess I was way off there.
By "printed" on a single line I mean: If there is an error with the code, or it's a test case that throws an assert, or a panic is thrown, etc..., all of those issues are shown to be on a single line, the macro calling line, according to the printout in the terminal window, regardless of the number of lines of this newly created block of code.
I am truly limited to what crates and binaries I have access to (internal company issues).
But thank you everyone. I'm just happy the macro is working.
This means that the Span of the tokens corresponding to erroneous code refer to the macro invocation token. They might also refer to some input tokens - then the error would point on them, too.
Yes. They are completely disregarded. The display impl just slavishly produces single spaces between tokens.
It wouldn't even be possible to take spans into account when printing. First of all, generated token streams can have no (useful) span info (ie., Span::call_site()). Furthermore, an existing token stream's contents can be modified in a way that the span information can't possible match the result, eg. when you replace a code section with a shorter or longer one than the original.
@H2CO3 darn, but I understand. Maybe I'll attempt again an Attribute Macro instead, to leave the developers code block where it is and just add the changes I want above the code block. I'm fighting against Androids Soong build system here.
Thank you everyone.
I have written code before for converting a general (ie. not necessarily Rust) token stream into a not-terrible string representation. It's a simple implementation, you can copy the whole thing if you wish.
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.