Mixing Optional and Non Optional parameter in macro

[Clement-Jean]

For the testing part of my lexer, I came up with a simple macro that let met define the expected token type (enum) and the token literal (string):

macro_rules! token_test {
	($($ttype:ident: $literal:literal)*) => {
        {
            vec!($($ttype,)*).iter().zip(vec!($($literal,)*).iter())
        }
    }
}

and then I can use it like this:

for (ttype, literal) in token_test! {
    Let: "let" Identifier: "five" Assign: "=" Int: "5" Semicolon: ";"
} {
    //...
}

However, this is a little bit verbose and we don't need to specify the literal for most of the token since I have another macro that transforms an enum variant into a string (eg: Let -> "let").

So what I hope to do is something like:

for (ttype, literal) in token_test! {
    Let Identifier: "five" Assign Int: "5" Semicolon
} {
    //...
}

And if I understood properly, I can use optional parameters to match either TYPE: LITERAL or TYPE (not sure how yet). Maybe something like:

macro_rules! token_test {
    ($($ttype:ident$(: $literal:literal)?)*) => {
        {
            //...
        }
    }
}

So then my question is is there a way to build Vector out of this?

To be more clear:

• In the case of no literal passed, it should add the string representation of my enum (eg: Let -> "let")
• In the case of literal passed, it should add the literal directly

[steffahn]

(I'm in mobile, so this is rather concise.)

You could pass it on to a different macro that expands to the expression.

….zip(vec![$(other_macro!($ttype$(: $literal)?),*])

Then define other_macro using two distinct cases/arms.

[Clement-Jean]

While waiting for my post to be approved (I'm new here), a couple of SO users and I solved the problem in a way that is pretty similar to what @steffahn is proposing.

We came up with this (any improvement welcomed):

macro_rules! token_test {
  (@some_or_none) => { None };
  (@some_or_none $entity:literal) => { Some($entity) };
  ($($ttype:ident $(: $literal:literal)?)*) => {
    vec!($($ttype,)*)
      .iter()
      .zip(vec!($(
        token_test!(@some_or_none $($literal)?)
          .unwrap_or($ttype.as_str().unwrap())
      ),*))
  };
}

but with @steffahn suggestion, I came up with something like:

macro_rules! token_test {
  (@ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
  (@ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
  ($($ttype:ident $(: $literal:literal)?)*) => {
    vec!($($ttype,)*)
      .iter()
      .zip(vec![$(token_test!(@ttype_or_literal $ttype$(: $literal)?)),*])
  };
}

[Yandros]

You can also do:

macro_rules! token_test {(
    $(
        $ttype:ident $(: $literal:literal)?
    )*
) => ({
    ::core::iter::zip(
        ::std::vec![
            $(
                $ttype
            ),*
        ],
        ::std::vec![
            $({
                let _f = || $ttype.as_str(); $(
                let _f = || $literal;        )?
                _f()
            }),*
        ],
    )
})}

Some extra remarks:

• Do these need to be vec![...]s necessarily? Couldn't they be [...]s arrays?

• Do you need to zip two such iterators, or could a simple array-of-pairs suffice?

[Clement-Jean]

Interesting remarks! I'm actually just looking for something that I can iterate over and can be deconstructed into (type, literal). I'm interested to see what you have in mind

[Yandros]

In that case:

macro_rules! token_test {(
    $(
        $ttype:ident $(: $literal:literal)?
    )*
) => (
    [$(
        (
            $ttype,
            {
                let _f = || $ttype.as_str(); $(
                let _f = || $literal;        )?
                _f()
            },
        )
    ),*]
)}

ought to do it (edition 2021)

[Clement-Jean]

Amazing! I implemented this with the two branches mentioned previously and I have something like:

macro_rules! token_test {
  (@ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
  (@ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
  ($($ttype:ident $(: $literal:literal)?)*) => {
    [$(($ttype, token_test!(@ttype_or_literal $ttype$(: $literal)?))),*]
  };
}

[matt1992]

Dropping this in the mix of solutions:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=858bd6e3fe5de84a5dd1949c00c525f1

macro_rules! ignore_second {
    ($value:expr $(, $_ignored:expr)? $(,)?) => { $value }
}

macro_rules! token_test {
  ($($ttype:ident $(: $literal:literal)?)*) => {
    [$(($ttype, ignore_second!($($literal,)? $ttype.as_str().unwrap()))),*]
  };
}

#[derive(Debug, Copy, Clone)]
struct Bytes(&'static [u8]);

impl Bytes {
    fn as_str(self) -> Option<&'static str> {
        std::str::from_utf8(self.0).ok()
    }
}

fn main(){
    let foo = Bytes(b"foo s");
    let baz = Bytes(b"baz s");
    let qux = Bytes(b"qux s");
    println!("{:?}", token_test!(foo: "bar" baz qux: "hello"));
}

this is a trick I use in a bunch of macros, since it allows defaults without multiple macro arms, so there can be many arguments with defaults in the same macro invocation.

[Clement-Jean]

Oh wow, not sure I understand everything, I'll need to play with it a little, but this seems to be taking a different approach where the type will be ignore if the literal exists. Smart one !

Yup, in a compact way, I made:

macro_rules! token_test {
  (@ignore_second $value:expr $(, $_ignored:expr)? $(,)?) => { $value };
  ($($ttype:ident $(: $literal:literal)?)*) => {
    [$(($ttype, token_test!(@ignore_second $($literal,)? $ttype.as_str().unwrap()))),*]
  };
}