Tokenstream -> tokentree ->?

Is there something "higher" than tokentree ?

pub struct Foo { ... }

becomes an Iter<TokenTree> of

Ident { pub }
Ident { struct }
Ident { Foo }
Group { ... }

conceptually, this is "a struct", but it is spread out over 4 tokentree, and even more if we do pub struct Foo<...> { ... }

Is there something 'higher level' than a tokentree, where this is conceptually higher level, instead an iterator of "token tree"s ?

You have already been recommended the syn crate for parsing Rust code into an actual AST in a previous post. A struct is, unsurprisingly, represented by an ItemStruct.

3 Likes

A token stream is a sequence of token trees, there are intentionally no other concepts represented directly in procedural macros.

2 Likes

Yes, so my question is: is there something higher level like 'AstTree" or something, and a way to do Iter<TokenTree> to a hypothetical Iter<AstTree>

You can do whatever you want inside your macro, but you have to convert it all back into a token stream to output it.

I'm not aware of any existing "medium" abstractions like what you're asking for (though I'm sure some exist). The complexity of manually dealing with tokens at all is usually not worth it so macros just pull in syn so they don't have to deal with it. Then then can work with constructs that more closely resemble Rust code as they're used to thinking about it

2 Likes

I would recommend creating your own struct and implementing quote::ToTokens for it. That's probably the best mid-level abstraction for code generation purposes.

I wouldn't normally reach for syn::ItemStruct because it's too verbose to construct one manually and parsing from a TokenStream doesn't actually give you anything other than detecting syntax errors that rustc will tell you about anyway.

I think we are solving two different problems.

In a previous post, I asked about the following problem:

  1. I have some Abstract Data Types
  2. I manually express some data using these ADTs.
  3. I need to write a rust function g: ADT -> String (rust code)

Your advice makes perfect sense for writing function g.

Since that previous post, my problem has changed about (though not explicitly stated in this new question). My problem now is:

  1. Squinting a little, there is a bijection of ADT <-> subset of Rust code
  2. Expressing data in this ADT is a bit tedious.
  3. I am now working on two functions:
f: String (Rust source) -> ADT
g: ADT -> String (Rust source)

so instead of writing ADT manually, I just write in some restricted subset of Rust (with all of IntelliJ's IDE support).

For this function f, syn::ItemStruct turns out to be useful.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.