Recursive macro expanding same inputs to different outputs

I created a simple macro that converts any nested array of numbers into a recursive enum of vectors.

#![feature(trace_macros)]

pub enum VecEnum {
  Vec(std::vec::Vec<VecEnum>),
  Entry(f64),
}

#[macro_export]
macro_rules! array_to_vec {
  // Match arrays of arrays of expressions
  ([$([$($x: expr),*]),*]) => {
    // Outer array becomes a vec. Inner array is expanded recursively.
    VecEnum::Vec(
        vec![$(array_to_vec!([$($x),*])),*]
      )
  };
  // Match arrays of expresions
  ([$($x: expr),*]) => {
    VecEnum::Vec(
      vec![
        $(array_to_vec!($x)),*
      ]
    )
  };
  ($x: expr) => {
    VecEnum::Entry($x)
  };
}

#[test]
fn test() {
  trace_macros!(true);

  let good = array_to_vec!([1.0]);
  let bad = array_to_vec!([[[1.0]]]);

  trace_macros!(false)
}

Using trace macros I can see some strange behavior happening.

On the first input above, it works as expected.

However, on the second input it fails to compile and the macro trace shows it expanding the SAME input as before but with a different output. Why is this happening? Thanks for the help.

Playground

Explanation

Once something gets captured into a $_:expr metavariable, then emitting that metavariable does not yield exactly the source code that was captured by it: instead, it emits it but wrapped within "invisible" parenthesis. In the remainder of this post, I'll be using ⦑ ⦒ for such parenthesis:

macro_rules! array_to_vec {
  // Match arrays of arrays of expressions
  ([$([$($x: expr),*]),*]) => {
    // Outer array becomes a vec. Inner array is expanded recursively.
    VecEnum::Vec(
        vec![$(
-           array_to_vec!([$( $x ),*])
+           array_to_vec!([$( ⦑$x⦒ ),*])
        ),*]
    )
  };

Thus, if $x happened to be, itself, a [ … ] expression, when recursing,
your macro will stumble upon ⦑ [ … ] ⦒ rather than [ … ],
hence failing the first two rules and directly falling back to the third one, the one expecting a $_:expr.

Hence those last two lines you've highlighted.

See also:

Or the README of ::defile, a helper proc-macro to palliate this limitation.

Solution

  • defile - Rust would be one solution;

  • in your case, however, I think a simpler one is to simply treat the innards of the array as an "opaque" blob of $:tts that you shall forward verbatim:

    #[macro_export]
    macro_rules! array_to_vec {
        // Match arrays of arrays of expressions
        (
            [
                $(
                    [
    -                   $( $x:expr ),* $(,)?
    +                   $($x:tt)*
                    ]
                ),* $(,)?
            ]
        ) => (
            // Outer array becomes a vec. Inner array is expanded recursively.
            VecEnum::Vec(
                vec![$(
    -               array_to_vec!( [$($x),*] )
    +               array_to_vec!( [$($x)*] )
                ),*]
            )
        );
    
        // Match arrays of expresions
        (
            [
                $( $x:expr ),* $(,)?
            ]
        ) => (
            VecEnum::Vec(
                vec![$(
    -               array_to_vec!($x) // Not necessary, but it makes thing clearer (this recursion would always hit the third rule)
    +               VecEnum::Entry($x)
                ),*]
            )
        );
      
        (
            $x:expr
        ) => (
            VecEnum::Entry($x)
        );
    }
    
1 Like

Thanks so much for this detailed explanation. Much obliged :pray:

1 Like