Why macro_rules! scoping is so confusing?

I don't understand why macro_rules! visibility is so complicated. Maybe there is some book that can describe it but I couldn't find it (so far the best source is Scoping but still lacking).

This is what I'm talking about:

These scoping rules are why a common piece of advice is to place all macros which should be accessible "crate wide" at the very top of your root module, before any other modules. This ensures they are available consistently.

More to the point why can't macro_rules! be referenced like a regular "functions" from any module by any module (if visibility allows) and have instead follow very complicated/different rules of visibility... All those [macro_use], [macro_export], macro something.

I know there must be a reason for this but comparing to ease of use of defines in C (yes, they are somewhat different) macro_rules! are just very not user friendly.

2 Likes

Absolutely. C macros are "easier" to use and easier to screw up.
As for why macros have a different scoping rule than other things is quite simple - macros aren't an "item" as far as the language is concerned (for what is an item, ref to the Rust reference). Macros don't technically "belong" to a module. The code of a macro is generated where it is called, and not where it is defined.
Of course, the reason I have mentioned is an intuitive one. There may be technical difficulties in making macros accessible using the same rules as everything else - but I don't know.

1 Like

I understand all of those but they don't describe why scoping should be so complicated...
Ok, they are not "technically" belong to a module, but they are located there.
Ok, they are not an "item" as far as the language is concerned, but for developers they are still a callback item.

What I'm talking about is that it would be so much easier if they were imported together with the module they are in. Modules are basically just a path to the item anyways.

Because macros are expanded so early in the compilation process, much of Rust's standard visibility system is inaccessible. For example, macros are allowed to define new modules, so the module tree is necessarily missing or incomplete when macros are processed.

8 Likes

macro_rules! mostly still uses the old legacy scoping rules. In Rust 1.0, the more complicated (to implement) name resolution for macros wasn't yet implemented, so this was the only way to import macros. Nowadays you can at least reexport macros at any point in the module hierarchy, and also import them from other crates with use. In Rust 1.0 neither of those were possible.

It isn't unlikely that macro scoping will eventually work like scoping for all other items (perhaps in some future edition).

9 Likes

I was talking only about accessing defines. They follow same rules as everything else. How defines work, define guards and other stuff is a completely different story and is not part of my question.

That is not how they work even today. Macro expansion and name resolution (which computes and uses visibility rules) are done at essentially the same time in the compilation process.

The compiler is perfectly capable of resolving a path to a macro, and then expanding it, creating more items (procedural macros and "2.0" macros already work like this, and macro_rules! could, in the future).

Rust editions can make limited breaking changes to the language, they could be used to transition macro_rules! to the newer model. "2.0" macros already use it today, but are still unstable.

Not really. Simple example:

int foo() {
   int scope;
   #define SET_SCOPE(x) scope = x
   return scope;
}

int main() {
    int scope;
    SET_SCOPE(42);
    printf("%d\n", scope);
}

Note how I can not access foo's scope variable (I need my own). Note how SET_SCOPE can easily access scope variable in main.

Many years ago, when computers were big but memory was small macros in C were handled by a different binary, before actual C compiler was called! How can they follow scope rules with that approach?

They ignore scopes in C, too.

Just verified: you are correct. Hmm. Proc macro can afford that because they come from a different crate thus can not both create entities and reference them. "2.0" macros can not create new items (which makes them pretty much useless: how would you do something like nonzero_integers! with them?) it would be interesting to see how they plan to resolve that dilemma without “macro must be declared before it can be used” rule.

But yeah, you can correct, if "macros 2.0" would ever evolve to the state where they would be actually usable then rust edition mechanism can be used to switch.

Again, I was talking about accessing the marcros itself (the process of actually calling the damn thing) and nothing about generated content.

They can, though

#![feature(decl_macro)]

macro m() { struct S; }

m!();

Again, this is already solved, since macro expansion and name resolution happens interleaved. You can already try it out, too, because a #[macro_export] macro is reexported at the crate root using non-legacy scoping, so this compiles fine:

mod m {
    use crate::mac;
    
    mac!();
}

#[macro_export]
macro_rules! mac {
    () => {
        pub struct S;
    }
}

use m::S;

Bad wording, I guess. Sorry. Yes, technically, you can create new useless items, but what's the point? Have you actually created anything new if noone outside can see that?

That is how you create items:

macro_rules! m { () => { struct S; } }

m!();

fn foo(s: S) -> S {
    s
}

With macro 2.0… doesn't work.

And if you would make them, you know, actually useful, then you would open the Pandora's box: who may access items created in macro? When? How? What if item created by macro A requires item created by macro B and macro B requires item from macro A? What if they conflict?

So many interesting ways to screw the whole system… yet, as I have shown, even standard library needs that, without such ability “macro 2.0” is not very useful.

This is "hygiene", which is absolutely the hardest part of making a macro system. macro_rules are only sortof-hygienic; "Macros 2.0" is trying to be more principled. But yes, the details are not yet determined.

1 Like

Your example compiles fine but this is not what you need for "macro 2.0".

If you complicate this example a bit then it stops compiling.

Version which doesn't try these fancy new scopes works fine, of course.

And that's kind of the issue. “macro 2.0” are trying to revolutionize the way macro system works. And may actually succeed and create another revolution (like borrowing system did a revolution in languages design and enabled “safety without GC”).

But at this point I'm highly skeptical: the fact that you can reexport macro and then import it already create quite a mess (why was it done? was it actually useful enough to support?), do you really want to create rules which would be either useless (like “macro 2.0” are today) or strange and somewhat bizzare? Can you, as the bare minimum, make this thing work, somehow:

mod m {
    use crate::mac;
    use crate::cam;
    
    cam!();
    mac!();
}

#[macro_export]
macro_rules! mac {
    () => {
        #[macro_export]
        macro_rules! cam {
            () => {
                pub mod foo {
                    pub struct S;
                }
            }
        }
    }
}

use m::foo::S;

I have tried to change it in different ways but haven't found how I can make it work. Again: simple and straightforward macro_rules work without a hitch (without obvious and very easy to understand limitations, of course).

What do you mean by "work" here? As far as I understand, this code is meaningless because it attempts to import crate::cam which does not exist (and that is the error I see on the playground). Are you saying nested macros should be defined even when the outer macro is never invoked?

See also:

mod foo {
    macro_rules! bar {() => ()}
    pub(crate) use bar;
}

foo::bar!(); // Works!

Regarding the discussion about decl_macros, the feature has to do with

full hygiene

It has the advantage that when you call a macro macro, a caller knows the kind of items that may be affected by it. That's the whole point of hygiene:

struct S;
mod foo {
    use super::*;

    ::some_crate::some_macro!();

    fn foo (s: S) {} // <- no matter what `some_macro!` expands to, thanks
                     //    to hygiene, we *know* `S` refers to what we wanted.
}

This is the whole point of hygiene. Current macros, be it procedural ones, or macro_rules! ones, can't guarantee this, and thus one ends up writing a lot of __VAR code, and ::core::result::Result::Ok kind of paths, again, because of hygiene. And let's not forget those very pretty #[doc(hidden)] pub mod __ { modules to provide internal helpers to one's macros.

All these things are solved by the proper hygiene of macro macros.

macro m {( $StructName:ident ) => (
    struct $StructName;
)}

m!(S);
const _: () = {
    let _: S; // OK
};

Just Works :tm:, and so does:

pub
macro m {() => (
    const _: () = {
        // ...
        let _: private::MacroHelper;
    };
)}

// private! No need to `#[doc(hidden)] pub`
mod private {
    pub
    struct MacroHelper;
}

So, granted, sometimes we still want unhygienic behavior. At that point it is unfair to criticize a partially implemented feature: there is indeed currently no way to opt-out of the full hygiene it imbues its defined elements with (except by making the macro take extra identifier parameters, as showcased above), but there will likely be a way:

/// Imaginary syntax
macro m {() => (
    struct #S;
)}
// or, another possible syntax:
match! (S) {( $S:ident ) => (
    macro m {() => (
        struct $S;
    )}
)}

All in all, macro macros improve the situation quite a bit.

Their only drawback, right now, is that their shorthand sugar macro m() { ... }, which, you may have noticed, I have tried not to use, does not wrap the expanded stuff in extra braces. So they will still be quite confusing for beginners trying to write:

macro m() {
    let x = 42;
    x
}

let y = m!(); // Error

instead of:

macro m {() => ({
    let x = 42;
    x
})}
3 Likes

As far as I understand, this code is meaningless because it attempts to import crate::cam which does not exist (and that is the error I see on the playground).

That's what I see on playground, too. But then the question is: why it doesn't exist? mac! called in the next line should create it and make it accessible from the crate root (or maybe m? tried that, too) … then why that haven't happened, hmm?

And yes, if I move mac! explansion into a different place (without changing it's definition or anything else!) — it works.

IOW: rules which macro_rules! follows may not be prettiest and nicest in existence, but they are simple to understand, unambigous and useful.

Rules created by “macro 2.0” effort (and attempt to use prematurely them with #[macro_export]) are complex, ambiguous and confusing (and “macro 2.0” today are not very useful in addition to all that).

And it's not entirely clear what problem they are solving (no, the need to move macro definitions sometimes is not a huge problem which needs immediate solution in practice).

Maybe one day they would provide something good. Maybe. But I'm not holding my breath.

#[macro_export] is not an "attempt to use prematurely" macros 2.0. #[macro_export] is an old method for exporting macros from before macro_rules! worked the way it does now (I believe #1561 is the relevant RFC; note #[macro_export] already existed at that point). So it is not clear what you are actually criticizing about macros 2.0, since the hygiene issue is explicitly incomplete (although I doubt it is as serious a problem as you seem to think; hygiene is generally desirable for the reasons described by Yandros) and the #[macro_export] issue is completely irrelevant.

With the change to crate::m::cam I think I see what you are going for with the example, and I completely agree the combination of macro_rules! and macro_export is confusing and inconsistent. But that is to be expected, given the messy history; I have no idea why you think macros 2.0 will make things worse.

1 Like

Well… this discussion made me even more skeptical about “macros 2.0”, surprisingly enough.

Because I haven't looked on Rust too deeply while it wasn't mature enough to write stable code in thus I don't even remembered if macro_rules! have come before #[macro_export] or after.

But looking on the state of affairs today I observe that macro_rules! is just a very tiny (yet logical) improvement on top of venerable C macro system which was continuing from from earliest autocoder systems.

And comparable to how ANSI/ISO C improved upon previous macrosystems by switching from text-expansion system to token-expansion system (adequate improvement for 30 years of development, don't you think?) macro_rules! improved that by switching to token trees (and hey, they even have that funny property of C++ macros where language pairs angle brackets, but macrosystem doesn't) and adding a tiny bit of hygiene on top.

Which looked like a nice, sane, improvement for another 30 years of development.

#[macro_export] complications (when used inside of one crate, not cross-crate) looks like mostly pointless complication and, basically, a solution in a search of a problem and “macros 2.0” even more so.

Now, when I recalled that macro_rules! weren't created in an obvious process of combining ANSI/ISO C macros with venerable M4 macros, but in the process of simplifying overly-complex and hard-to-use syntax extensions system of early Rust… I wonder even more: why do you think Rust even need something radically different? After all that experimentation we, basically, returned back to where we were 30 years ago with a minor improvement. Is it really a coincidence?

What actual problem would be solved by “macros 2.0” ? Because I've looked on many proposals and they all look like a solution in a search of a problem. As in: they solve lots of imaginary problems (like: macros are not first-class citizen in the Rust module system — but why is that a problem?macros cannot be imported or named using the usual syntax — but why is that a problem and not the advantage?we have to use things like ::core::result::Result::Ok in public macroses — but is it such a big problem in practice?) but where exactly they improve the situation quite a bit?

What crate? What code?

I mean: yes, I understand that feature is still unfinished, but it's very-very significant feature and usually articles in blogs which extol virtues of something like that are published way before feature is stabilized. Like was with const generics and GATs. And they usually show why new feature allows one to do something that's either impossible or infeasible to do with existing features.

So… what exactly is impossible or infeasible in Rust without “macros 2.0”?

That's... isn't that exactly the problem that your code example from earlier demonstrates? I.e., you can't use a macro with the normal rules depending on where it's defined, precisely because macros aren't items and they aren't processed properly out of order like other items are, nor do they work with paths and visibility in the same way as everything else. In other words the problems you are saying are with macros 2.0 are in fact exactly what's wrong with macros 1.0 that the proposals are trying to fix.