How to understand "the second macro will see an opaque AST" in the reference?

[Yandros]

There is no bug here, it's all working as intended. I've explained this phenomenon several times already, posting it here for reference:

@Yandros

The subtle thing here is that the macro metavariable / transcriber captures which are not :tt , :ident , or :lifetime actually wrap the captured syntax within "invisible parenthesis" that make it no longer be something as simple as an identifier: the "real" expansion of that with_u32!( u32 ) call, with our invisible parenthesis goggles on, is:

   ⁽u32⁾ :: to_be_bytes ( some_expression )
// ^^^^^
// a type

which does not match the syntax PathInExpression syntax at all, and doesn't match the QualifiedPathInExpression syntax by that tiny detail of there not being the necessary angle brackets.

Hence why adding the angle brackets was necessary (a :ty macro metavariable/transcriber (such as ⁽u32⁾ ) is obviously a valid Type within Rust grammar).

Note that these "invisible" parenthesis are not shown on cargo expand / rustc -- … {un,}pretty=expanded output, hence why this error message may look especially confusing given that one can copy-paste the output of such expansion and the code will work.

In some cases, this is a reason to use $($T:tt)* to capture the $T as an arbitrary sequence of arbitrary tokens which are later on emitted verbatim , causing no issues except when generics parameters appear (in which case @Hyeonu's remark about a necessary turbofish syntax or <…> wrapping applies). But then we have the issue of it being quite hard to parse multiple such $T s, since nothing can follow a :tt)* repetition, and thus each type needs to be grouped within parenthesis, brackets, or braces, to avoid ambiguity errors:

macro_rules! with_types {(
    $( ($($T:tt)*) )*
) => (
    $(
        let _ = $($T)* :: to_be_bytes ( some_expression );
    )*
)}
with_types![
    (u32)
    (::core::primitive::u64)
    (Vec<String>) // Error, `Vec < String` expression followed by extraneous code
];

A more general solution to this issue is, since proc-macros can see and interact with those invisible parenthesis, to use a helper proc-macro which can strip them, such as:

GitHub - danielhenrymantilla/rust-defile: Helper proc-macro to "ungroup" a captured metavariable

---

Macro_rules matching different types

Also, regarding the initial question, once a macro input is captured in a metavariable whose kind is not :tt nor :ident, then it gets "wrapped in invisible parenthesis" which prevents it from matching any other literal rule. foo!( bool ) receives the sequence of tokens `bool`, which can be parsed as a :ty and captured within one, as $tp := `⦑bool⦒`, (I am using ⦑ and ⦒ to represent the "invisible parenthesis"), which is then fed to an invocation of foo!(_ $tp), i.e., foo!( _ ⦑bool⦒ ), which r…

---

@Yandros

Why the nested macro call fails

This is due to "invisible grouping" / wrapping in "invisible parenthesis": this is something which happens to all macro captures / metavariables / transcribers but for :tt , :ident , and :lifetime . So, in your case, once you capture, say, Option<i32> as a :ty , everytime you emit that metavariable, you are actually emitting ( Option<i32> ) , with invisible and ty pe-tagged parenthesis. And ( ... ) won't thus match the $id:ident case you had, but the $other:tt fallback.

Quoting The Reference:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident , lifetime , and tt fragment types are an exception, and can be matched by literal tokens. The following illustrates this restriction:

macro_rules! foo {

($l:expr) => { bar!($l); }

// ERROR: ^^ no rules expected this token in macro call
}

macro_rules! bar {
(3) => {}
}

foo!(3);

For the general case, you can use:

GitHub - danielhenrymantilla/rust-defile: Helper proc-macro to "ungroup" a captured metavariable

to work around that restriction. But using a proc-macro helper can be deemed a bit too heavy-weight, and is indeed not necessary in your case.

Some workarounds to palliate your issue

Rather than capturing a :ty , you can (sometimes) capture a $($_:tt)* repetition

---

@Yandros

Explanation

Once something gets captured into a $_:expr metavariable, then emitting that metavariable does not yield exactly the source code that was captured by it: instead, it emits it but wrapped within "invisible" parenthesis. In the remainder of this post, I'll be using ⦑ ⦒ for such parenthesis:

macro_rules! array_to_vec {
  // Match arrays of arrays of expressions
  ([$([$($x: expr),*]),*]) => {
    // Outer array becomes a vec. Inner array is expanded recursively.
    VecEnum::Vec(
        vec![$(
-           array_to_vec!([$( $x ),*])
+           array_to_vec!([$( ⦑$x⦒ ),*])
        ),*]
    )
  };

Thus, if $x happened to be, itself, a [ … ] expression, when recursing,
your macro will stumble upon ⦑ [ … ] ⦒ rather than [ … ] ,
hence failing the first two rules and directly falling back to the third one, the one expecting a $_:expr .

Hence those last two lines you've highlighted.

See also:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident , lifetime , and tt fragment types are an exception, and can be matched by literal tokens.

• link

Or the README of ::defile , a helper proc-macro to palliate this limitation.

Solution

• defile - Rust would be one solution;
• in your case, however, I think a simpler one is to simply treat the innards of the array as an "opaque" blob of $:tt s that you shall forward verbatim

---

---

@wheird_lee

Did I misunderstand what has been explained in the reference? if I did, how to understand it correctly?

From your first post, it looks like you got the :tt / :ident vs. matcher situation reversed:

• If foo! were to capture a :tt, :ident (or :lifetime), then you'd be able to transparently match against the exact token capture within bar!.

• But if foo! uses a high-level auto-grouped capture such as :expr, then that metavariable will thenceforward represent an invisibly-parenthesized group. It's thus a single token tree (quite handy for recursing, btw), but one which appears opaque to the second macro, in the same fashion that bar! { ( quux ) } will not be a valid call if bar! were to expect a quux argument.

  So, bar! can only handle a higher-level-grouped metavariable from foo! if and only if:

  • It takes a :tt, since all (parenthesized, braced, …) groups, including the "invisibly parenthesized" ones, are single token-trees each, or if it takes some other higher-level capture compatible with the first one: an :expr is compatible with an :expr (and more generally, for any kind, a :kind is always compatible with :kind), and then you can have a :path be compatible with :expr, or with :ty, or with :pat; an :item is compatible with a :stmt , a :block is compatible with an :expr, etc.

  See what the reference has to say about the Rust grammar to better figure out these compatibilities.