Splitting a new thread for this, since it isn't anything to do with either the original question or the meta-discussion that the thread turned into.
I was talking about how a macro-by-example's arguments must all come from the single brace, bracket, or parenthesis group that immediately follows. You can't get around that with tt-munching as far as I know.
macro_rules! itself is an example of the sort of thing you can't do with macros by example. I'd really like to be able to write macros that are invoked like
which is the least ugly workaround I could think of in the actual code that inspired this example. And I'd also really like to be able to write macros that look syntactically like control structures:
repeat! {
// loop body
} until /* controlling expression */ ;
This implies that the compiler needs to know, before finishing name resolution, that the use statement is actually an import and not part of the macro invocation.
As a historical note, macro_rules! used to be parsed with the same macro parsing machinery, and it was possible (unstably) to parse (e.g. in cfg-excluded items) the syntax with other bang identifiers. Today, however, macro_rules! is treated as its own syntax kind, and is only a conditionally reserved name; it's allowed to define a macro_rules! macro_rules if you're so inclined.
I know. I think that was an incorrect design decision, and furthermore, one that has been thoroughly undermined by proc macros, to the point where the only thing we accomplish by continuing to insist on it is to force people to use proc macros when they shouldn't have had to.
I see no good reason why declarative macros 2.0 should not be at least as capable as Scheme's syntax-case. (What we have now is not even as powerful as the more limited syntax-rules despite having very obviously been inspired by syntax-rules.)
Blech. I can think of workarounds, but honestly my recommendation would be to require macro! macros to be declared textually prior to use. It makes them different from other kinds of items, but in a way that's easy to explain and motivate (using exactly this sort of example).
Assuming use bitflags::bitflags has appeared earlier in the file, the parsing of these examples would be almost entirely up to the definition of bitflags::bitflags, just as it is now for procedural macros.
I would draw only one hard line: a macro invoked inside a scope should not be able to consume the closing delimiter for that scope. So, in your second example, bitflags::bitflags! might eat the let _ = 42 but it would definitely not eat the close brace for the function definition or anything after that.
I think you're only technically correct with respect to derive and attribute macros. For example, I had been under the impression that
repeat! {
// loop body
} until /* controlling expression */ ;
was currently possible with proc macros, and what you're saying means it's not ... unless there is an enabling attribute macro applied to the fn item. The syntax of items is so general that, in practice, if you have
#[proc_macro]
fn name (...) -> ... { ... }
the syntax glossed over by each of the ... is whatever the proc macro wants it to be.
This is such a weak constraint it might as well not exist.
attribute macros can do anything to the parsed item, for example, #[tokio::main] requires an async fn item, but then removes the async qualifier to allow main() to be called by the stdlib startup code.
An attribute macro cannot make your example repeat! {} until expr; valid syntax, because while repeat! {} will parse as a block that’s a macro call, until expr is invalid.
For example, if we use the #[cfg] attribute to entirely disable some code, it still won't parse:
error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found keyword `true`
--> src/lib.rs:3:25
|
3 | repeat! { 1 } until true;
| ^^^^ expected one of 8 possible tokens
Replacing cfg with an attribute macro won't make this code compile, no matter what the attribute macro does. Certainly there are lots of significant rewrites one could do to a function body that would appear to give macros in that body new capabilities, but I don't think “might as well not exist” is a reasonable description.
Hm, OK, I stand corrected. I don't think that really changes my point, though. Proc macros can take in almost any input syntax, and can emit completely arbitrary code as long as it's well-formed. In my view this means there isn't any good reason to keep macros-by-example as limited as they are, because if people can't do what they want with a MBE and they're determined enough they will just use a proc macro instead. All we are doing by limiting MBEs is making those people do extra work.
(If it were completely up to me, proc macros would be allowed to take input syntax that the main parser doesn't know how to parse, with the only restrictions being that they can't mess with input tokenization or delimiter pairing or consume tokens past the end of the scope where they are invoked. But this is a separate issue from the limitations of MBEs.)
Cannot be used to define derive macros or attribute macros. This might be added in the future, as I linked in my previous post, and can be worked around for now at the price of some inconvenience in usage.
Not expressive enough to perform complex transformations. This is not a restriction on the input syntax; it cannot be solved by removing a constraint, but only by adding new mechanisms that don’t currently exist.
I feel like it's worth emphasizing that these restrictions aren't specific to declarative macros. Nor do I believe they're solvable without fundamental, catastrophic changes to rustc.
Very few languages allow extending the syntax in arbitrary ways, and for good reason. Not only is it incredibly difficult to do at all, it also makes parsing with anything but a full compiler nigh impossible (you can kiss your syntax highlighting goodbye). It also leads to obscure, esoteric syntaxes that are difficult for both the compiler and user to understand.
As to macros being "not worth it" without these features: the fact that almost every crate uses or provides some form of macro, speaks volumes as to their amazing utility. I myself have written many declarative macros that were comparable to proc macros in power.
There are several problems with this approach. How would it even be defined? How far can the unrecognized syntax happen for it to be given to the proc-macro? How are IDEs supposed to differentiate syntax errors from token supposed to be given to a proc-macro? And most importantly, how would Rust add any new syntax in a backward compatible way without breaking these macros?
MBEs also:
can't create new identifies;
are more restricted in the hygiene they can use.
However note that MBEs also have a superpower that proc-macros don't: $crate.