How to understand "the second macro will see an opaque AST" in the reference?

[wheird_lee]

As the reference explains in section `Transcribing` about macros by example:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident, lifetime, and tt fragment types are an exception, and can be matched by literal tokens.

In my understanding, any fragment specifier except whose type is one of ident, lifetime, and tt can only be matched in the second macro using a fragment specifier of the same type.

But this code works (playground link):

macro_rules! foo {
    ($l:expr) => { bar!($l); }
}

macro_rules! bar {
    ($l:tt) => {}
}

foo!(3);

So I'm wondering if it works using ident or lifetime instead of tt, but run this code I got an error (playground link):

/*--- snip ---*/

macro_rules! bar {
    ($i:ident) => {}
}

foo!(a);  // Error!

To my surprise, the first code still works if I change the type of $l from tt to literal:

/*--- snip ---*/

macro_rules! bar {
    ($l:literal) => {};
}

foo!(3);

---

Did I misunderstand what has been explained in the reference? if I did, how to understand it correctly? Or the reference makes a mistake?

[bradleyharden]

I believe the documentation is saying this will work

macro_rules! foo {
    (3) => {};
}

foo!(3);

But this will not

macro_rules! foo {
    ($l:expr) => { bar!($l); };
}

macro_rules! bar {
    (3) => {};
}

foo!(3);

The compiler seems to agree.

The documentation is saying that you can only match on the literal token 3 in the first macro. In the second macro, you must match it as a token tree, expression, code literal, etc. The second macro sees only an "opaque abstract syntax tree". It cannot see through the token type (expr, tt, literal) to see the actual contents of the token (in this case 3).

[wheird_lee]

But how to explain why this code doesn't work? (playground link)

macro_rules! foo {
    ($e:expr) => { bar!($e); }
    // ERROR:           ^^ no rules expected this token in macro call
}

macro_rules! bar {
    ($i:ident) => {}
}

foo!(a);

[2e71828]

The only match arm in the definition of bar requires an ident, but foo doesn’t have one of those. Instead, $e is an opaque expr which might be something other than a single identifier, such as a literal 42 or a compound expression a + b. The expansion rules forbid this regardless of what’s actually present at the callsite.

[Cerber-Ursi]

The main point of confusion is, I think, why didn't the compiler throw an error on the chain 3 -> $expr -> $literal (as in the third example by OP)? It probably is expected to fall under the same logic: $expr might not be $literal, so the inner macro call should fail, but it doesn't.

[2e71828]

My guess is that was originally a compiler bug, which turned into a special case to avoid breaking some pre-existing Rust code.

[wheird_lee]

I'm convinced there's a compiler bug. See the code below, it prints bar!(3 + 4) = 7, surprisingly! (playground link)

macro_rules! foo {
    ($e:expr) => { dbg!(bar!($e)) }
}

macro_rules! bar {
    ($($t:tt)?) => {
        $($t)?
    }
}

fn main() {
    foo!(3 + 4);
    // dbg!(bar!(3 + 4)); // error
}

[Cerber-Ursi]

Well, the last example is not very surprising, in fact, if you consider that the following compiles too:

macro_rules! bar {
    ($($t:tt)?) => {
        $($t)?
    }
}

fn main() {
    dbg!(bar!{ (3 + 4) });
}

(This, however, throws a warning about unnecessary parentheses, which is a false-positive - without them, as you've already found, the code would not compile. I'd probably search for the issue a little later and create it if it doesn't exist yet)

The point is that (3 + 4) is a single token tree. In procedural macros, it would be represented as TokenTree::Group. So the whole parenthesised expression is matched as $tt and passed as-is into the expansion.

The same thing applies to the already-matched items, such as $exprs: they are treated "as if" they are wrapped in the invisible parentheses - in procedural macros, this correspond to the Group with Delimiter::None; that's what allows this code to provide expected result:

macro_rules! double {
    ($x:expr) => {
        2 * $x
    };
}

fn main() {
    dbg!(double!(1 + 2)); // prints 6
}

If not for these invisible groups, double!(1 + 2) would expand to 2 * 1 + 2 (as it does in C) and evaluate to 5, not 6.
So, if something in an $expr, it is again a single token tree, no matter what was matched as this $expr before; and so it can be matched by $tt.

On the other hand, 3 + 4 by itself is not a single token tree. It can be matched by $($t:tt)*, as a sequence of three token trees - 3, +, 4; but not by a single $tt.

[Yandros]

There is no bug here, it's all working as intended. I've explained this phenomenon several times already, posting it here for reference:

@Yandros

The subtle thing here is that the macro metavariable / transcriber captures which are not :tt , :ident , or :lifetime actually wrap the captured syntax within "invisible parenthesis" that make it no longer be something as simple as an identifier: the "real" expansion of that with_u32!( u32 ) call, with our invisible parenthesis goggles on, is:

   ⁽u32⁾ :: to_be_bytes ( some_expression )
// ^^^^^
// a type

which does not match the syntax PathInExpression syntax at all, and doesn't match the QualifiedPathInExpression syntax by that tiny detail of there not being the necessary angle brackets.

Hence why adding the angle brackets was necessary (a :ty macro metavariable/transcriber (such as ⁽u32⁾ ) is obviously a valid Type within Rust grammar).

Note that these "invisible" parenthesis are not shown on cargo expand / rustc -- … {un,}pretty=expanded output, hence why this error message may look especially confusing given that one can copy-paste the output of such expansion and the code will work.

In some cases, this is a reason to use $($T:tt)* to capture the $T as an arbitrary sequence of arbitrary tokens which are later on emitted verbatim , causing no issues except when generics parameters appear (in which case @Hyeonu's remark about a necessary turbofish syntax or <…> wrapping applies). But then we have the issue of it being quite hard to parse multiple such $T s, since nothing can follow a :tt)* repetition, and thus each type needs to be grouped within parenthesis, brackets, or braces, to avoid ambiguity errors:

macro_rules! with_types {(
    $( ($($T:tt)*) )*
) => (
    $(
        let _ = $($T)* :: to_be_bytes ( some_expression );
    )*
)}
with_types![
    (u32)
    (::core::primitive::u64)
    (Vec<String>) // Error, `Vec < String` expression followed by extraneous code
];

A more general solution to this issue is, since proc-macros can see and interact with those invisible parenthesis, to use a helper proc-macro which can strip them, such as:

GitHub - danielhenrymantilla/rust-defile: Helper proc-macro to "ungroup" a captured metavariable

---

Macro_rules matching different types

Also, regarding the initial question, once a macro input is captured in a metavariable whose kind is not :tt nor :ident, then it gets "wrapped in invisible parenthesis" which prevents it from matching any other literal rule. foo!( bool ) receives the sequence of tokens `bool`, which can be parsed as a :ty and captured within one, as $tp := `⦑bool⦒`, (I am using ⦑ and ⦒ to represent the "invisible parenthesis"), which is then fed to an invocation of foo!(_ $tp), i.e., foo!( _ ⦑bool⦒ ), which r…

---

@Yandros

Why the nested macro call fails

This is due to "invisible grouping" / wrapping in "invisible parenthesis": this is something which happens to all macro captures / metavariables / transcribers but for :tt , :ident , and :lifetime . So, in your case, once you capture, say, Option<i32> as a :ty , everytime you emit that metavariable, you are actually emitting ( Option<i32> ) , with invisible and ty pe-tagged parenthesis. And ( ... ) won't thus match the $id:ident case you had, but the $other:tt fallback.

Quoting The Reference:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident , lifetime , and tt fragment types are an exception, and can be matched by literal tokens. The following illustrates this restriction:

macro_rules! foo {

($l:expr) => { bar!($l); }

// ERROR: ^^ no rules expected this token in macro call
}

macro_rules! bar {
(3) => {}
}

foo!(3);

For the general case, you can use:

GitHub - danielhenrymantilla/rust-defile: Helper proc-macro to "ungroup" a captured metavariable

to work around that restriction. But using a proc-macro helper can be deemed a bit too heavy-weight, and is indeed not necessary in your case.

Some workarounds to palliate your issue

Rather than capturing a :ty , you can (sometimes) capture a $($_:tt)* repetition

---

@Yandros

Explanation

Once something gets captured into a $_:expr metavariable, then emitting that metavariable does not yield exactly the source code that was captured by it: instead, it emits it but wrapped within "invisible" parenthesis. In the remainder of this post, I'll be using ⦑ ⦒ for such parenthesis:

macro_rules! array_to_vec {
  // Match arrays of arrays of expressions
  ([$([$($x: expr),*]),*]) => {
    // Outer array becomes a vec. Inner array is expanded recursively.
    VecEnum::Vec(
        vec![$(
-           array_to_vec!([$( $x ),*])
+           array_to_vec!([$( ⦑$x⦒ ),*])
        ),*]
    )
  };

Thus, if $x happened to be, itself, a [ … ] expression, when recursing,
your macro will stumble upon ⦑ [ … ] ⦒ rather than [ … ] ,
hence failing the first two rules and directly falling back to the third one, the one expecting a $_:expr .

Hence those last two lines you've highlighted.

See also:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident , lifetime , and tt fragment types are an exception, and can be matched by literal tokens.

• link

Or the README of ::defile , a helper proc-macro to palliate this limitation.

Solution

• defile - Rust would be one solution;
• in your case, however, I think a simpler one is to simply treat the innards of the array as an "opaque" blob of $:tt s that you shall forward verbatim

---

---

@wheird_lee

Did I misunderstand what has been explained in the reference? if I did, how to understand it correctly?

From your first post, it looks like you got the :tt / :ident vs. matcher situation reversed:

• If foo! were to capture a :tt, :ident (or :lifetime), then you'd be able to transparently match against the exact token capture within bar!.

• But if foo! uses a high-level auto-grouped capture such as :expr, then that metavariable will thenceforward represent an invisibly-parenthesized group. It's thus a single token tree (quite handy for recursing, btw), but one which appears opaque to the second macro, in the same fashion that bar! { ( quux ) } will not be a valid call if bar! were to expect a quux argument.

  So, bar! can only handle a higher-level-grouped metavariable from foo! if and only if:

  • It takes a :tt, since all (parenthesized, braced, …) groups, including the "invisibly parenthesized" ones, are single token-trees each, or if it takes some other higher-level capture compatible with the first one: an :expr is compatible with an :expr (and more generally, for any kind, a :kind is always compatible with :kind), and then you can have a :path be compatible with :expr, or with :ty, or with :pat; an :item is compatible with a :stmt , a :block is compatible with an :expr, etc.

  See what the reference has to say about the Rust grammar to better figure out these compatibilities.