Is there a way to "preview" macro-derived TokenStreams?

[Fabio]

I generally use VS Code with the rust-analyzer to write code. I am happy with it in general. But debugging procedural macros that create a TokenStream (typically to automate the implementation of some functions) has been very difficult. VS Code tells me what the underlying error in the output TokenStream is, but there is no way for me to "see" the token stream to identify what otherwise might be an obvious syntactic error.

Does anybody know of a way to "preview" the token stream resulting from #[derive(Whatever)]?

[jjpe]

IIRC Tokenstreams are Displayable, so you can just do something like println!("{}", tokenstream);.

Though if you want nicer formatting, you might want to consider using cargo expand, which is installable via cargo install cargo-expand.

[Fabio]

Oh that seems simple enough. Thank you.

I added this line in the derive crate:

#[proc_macro_derive(DerivedStuff)]
pub fn my_trait(input: TokenStream) -> TokenStream {
    let ast: syn::DeriveInput = syn::parse(input).unwrap();
    // ... lots of stuff ...
    println!("{}", &tokenstream); // ADDED THIS LINE
    tokenstream.into()
}

Back in the "main" library, I made a test on a struct that has #[derive(DerivedStuff)] fail so it would show the output. Not sure if this is more complicated than what you imagined, but I was flying blind and now I can see.

[jjpe]

This is pretty much what I had in mind.

BTW, you can also run cargo test -- <TEST_NAME> --nocapture to show test println output without altering the test.

[Fabio]

Thx. I forgot about --nocapture

[Yandros]

I recommend using cargo expand for the general case, and for the case of a proc-macro of your own, print can be very useful but, alas, doesn't yield such a nice looking output on its own (not formatted nor colored).

In that case, you can write your own helper pretty_print_tokenstream to palliate that, and it will yield cargo expand-looking code:

danielhenrymantilla/fix_hidden_lifetime_bug.rs/blob/v0.2.3/src/proc_macros/showme.rs#L4-L48

---

Regarding debugging macros more generally, see this summary of mine.

@https://users.rust-lang.org/t/trouble-with-z-macro-backtrace/61289/3?#bonus-other-macro-debugging-tips-3

Bonus: other macro debugging tips

There are several kinds of errors one can have when writing macros:

• If you are dealing with a complex recursive macro

  or even, just a chain of macro invocations, with a problematic invocation in the middle.

  In that case, I suggest trace_macros! be used:

  • Write trace_macros!(true); before the problematic macro invocation (with an optional trace_macros!(false); after it, to reduce the noise from tracing unrelated macros).

• If the macro expands to syntactically invalid / rejected-by-the-parser code:

  • trace_macros! can still be useful, although a big expansion will hardly be readable in that format;

  • You can use log_syntax! { … } to enscope the output of the macro:

      macro_rules! my_macro {(
          inputs…
    - ) => (
    + ) => (log_syntax! {
          long output…
    + })}
    - )}
    
    

  • Or, if it is a proc-macro, you can simply

    call {e,}println!(…) on the output TokenStream{,2}.

• The macro expands to semantically invalid code:

  • -Z macro-backtrace

    can indeed be useful here to help you pin-point the problematic code inside the macro definition;

  • Otherwise, a good old expansion of the emitted code will be very welcome. Ideally, this is done through:

    cargo expand
    

    should it be installed, else:

    RUSTC_BOOTSTRAP=1 cargo rustc -- -Z unstable-options --pretty=expanded
    

    In both cases replacing the macro invocation with the copy-pasted code may provide better-spanned / detailed error messages

    • When manually pasting the expanded code, the error disappears

      This means your macro has one of the two following issues:

      • Hygiene: you may be trying to have caller-provided code refer to an internal-to-the-macro / macro-defined variable, or vice versa. This currently only applies to local variables (and, technically, to lifetimes and $crate);

      • Grouping / metavariables. See ::defile's documentation or the last paragraphs of this section from the Rust Reference.

[Fabio]

Wow- thanks for the super detailed and helpful answer. I will try to work through it- cargo expand looks pretty useful.