For the testing part of my lexer, I came up with a simple macro that let met define the expected token type (enum) and the token literal (string):
macro_rules! token_test {
($($ttype:ident: $literal:literal)*) => {
{
vec!($($ttype,)*).iter().zip(vec!($($literal,)*).iter())
}
}
}
and then I can use it like this:
for (ttype, literal) in token_test! {
Let: "let" Identifier: "five" Assign: "=" Int: "5" Semicolon: ";"
} {
//...
}
However, this is a little bit verbose and we don't need to specify the literal for most of the token since I have another macro that transforms an enum variant into a string (eg: Let -> "let").
So what I hope to do is something like:
for (ttype, literal) in token_test! {
Let Identifier: "five" Assign Int: "5" Semicolon
} {
//...
}
And if I understood properly, I can use optional parameters to match either TYPE: LITERAL
or TYPE
(not sure how yet). Maybe something like:
macro_rules! token_test {
($($ttype:ident$(: $literal:literal)?)*) => {
{
//...
}
}
}
So then my question is is there a way to build Vector
out of this?
To be more clear:
- In the case of no literal passed, it should add the string representation of my enum (eg: Let -> "let")
- In the case of literal passed, it should add the literal directly
(I'm in mobile, so this is rather concise.)
You could pass it on to a different macro that expands to the expression.
….zip(vec![$(other_macro!($ttype$(: $literal)?),*])
Then define other_macro
using two distinct cases/arms.
3 Likes
While waiting for my post to be approved (I'm new here), a couple of SO users and I solved the problem in a way that is pretty similar to what @steffahn is proposing.
We came up with this (any improvement welcomed):
macro_rules! token_test {
(@some_or_none) => { None };
(@some_or_none $entity:literal) => { Some($entity) };
($($ttype:ident $(: $literal:literal)?)*) => {
vec!($($ttype,)*)
.iter()
.zip(vec!($(
token_test!(@some_or_none $($literal)?)
.unwrap_or($ttype.as_str().unwrap())
),*))
};
}
but with @steffahn suggestion, I came up with something like:
macro_rules! token_test {
(@ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
(@ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
($($ttype:ident $(: $literal:literal)?)*) => {
vec!($($ttype,)*)
.iter()
.zip(vec![$(token_test!(@ttype_or_literal $ttype$(: $literal)?)),*])
};
}
1 Like
You can also do:
macro_rules! token_test {(
$(
$ttype:ident $(: $literal:literal)?
)*
) => ({
::core::iter::zip(
::std::vec![
$(
$ttype
),*
],
::std::vec![
$({
let _f = || $ttype.as_str(); $(
let _f = || $literal; )?
_f()
}),*
],
)
})}
Some extra remarks:
-
Do these need to be vec![...]
s necessarily? Couldn't they be [...]
s arrays?
-
Do you need to zip two such iterators, or could a simple array-of-pairs suffice?
3 Likes
Interesting remarks! I'm actually just looking for something that I can iterate over and can be deconstructed into (type, literal)
. I'm interested to see what you have in mind 
In that case:
macro_rules! token_test {(
$(
$ttype:ident $(: $literal:literal)?
)*
) => (
[$(
(
$ttype,
{
let _f = || $ttype.as_str(); $(
let _f = || $literal; )?
_f()
},
)
),*]
)}
ought to do it
(edition 2021)
1 Like
Amazing! I implemented this with the two branches mentioned previously and I have something like:
macro_rules! token_test {
(@ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
(@ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
($($ttype:ident $(: $literal:literal)?)*) => {
[$(($ttype, token_test!(@ttype_or_literal $ttype$(: $literal)?))),*]
};
}
Dropping this in the mix of solutions:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=858bd6e3fe5de84a5dd1949c00c525f1
macro_rules! ignore_second {
($value:expr $(, $_ignored:expr)? $(,)?) => { $value }
}
macro_rules! token_test {
($($ttype:ident $(: $literal:literal)?)*) => {
[$(($ttype, ignore_second!($($literal,)? $ttype.as_str().unwrap()))),*]
};
}
#[derive(Debug, Copy, Clone)]
struct Bytes(&'static [u8]);
impl Bytes {
fn as_str(self) -> Option<&'static str> {
std::str::from_utf8(self.0).ok()
}
}
fn main(){
let foo = Bytes(b"foo s");
let baz = Bytes(b"baz s");
let qux = Bytes(b"qux s");
println!("{:?}", token_test!(foo: "bar" baz qux: "hello"));
}
this is a trick I use in a bunch of macros, since it allows defaults without multiple macro arms, so there can be many arguments with defaults in the same macro invocation.
Oh wow, not sure I understand everything, I'll need to play with it a little, but this seems to be taking a different approach where the type will be ignore if the literal exists. Smart one !
Yup, in a compact way, I made:
macro_rules! token_test {
(@ignore_second $value:expr $(, $_ignored:expr)? $(,)?) => { $value };
($($ttype:ident $(: $literal:literal)?)*) => {
[$(($ttype, token_test!(@ignore_second $($literal,)? $ttype.as_str().unwrap()))),*]
};
}