Why macro_rules! scoping is so confusing?

You may as well declare inability to have two mutable references “a problem”. Hey, most other languages out there allow it, why Rust doesn't want to do that?

No, that's not a problem. That's nuisance. It would only be a problem if you would explain how this makes certain “business tasks” (things which come from requirements outside of programmers desire) hard. And maybe even then the right way would be not to try to solve the issue everywhere uniformly, but to add some “escape hatches” (like how unsafe Rust have pointers and safe Rust have Arc/Rc/RefCell/etc).

So you are proposing to introduce huge change to the language and make people who are familiar with C/C++/MASM/etc macros (and now with macro_rules!, too) feel confused and [try to] make them learn new rules (which are so complex you feel the need to say that after five years of experiments we are not yet even ready to discuss them because they are unfinished) just because of some perceived inconsistency the existing rules bring to the language?

I am not sure how that may ever work. They sound different enough that conversion from macro_rules! wouldn't be trivial in many cases and don't bring enough benefits for people to care.

Things like these are [relatively easy] to bring into language when it's new and have few users. But if you have lots of users? Have python3 train-wreak of an upgrade taught you nothing?

The most you can realistically expect is to add a new syntax and then keep both of them for the indeterminate time. Similarly to how C++ needed more than 20 years to add optional modules support and would need, probably, 20 more to make include headers obsolete. If even then.

What do “macros 2.0” bring that is important enough to start such a process?

I'm sorry, I'm not able to understand what you're arguing here and I don't think this discussion is trending in a constructive direction.

It is maybe not so complicated if the ordering issue is explained.

Inside a crate I think macro visibility is simple, just not matching anything else in Rust.

Walk through the crate's files like the compiler would do it, from the top down: Go down into the first module you see (all the way), and when that's done go back up and continue down the next module.

A macro is visible if it's defined before it's used when the crate is visited in this order (and the module where the macro is defined needs to be marked #[macro_use]). The usual way to do it is to put the modules where macros are defined before all other modules.

I would try to summarize our positions and then we may agree to disagree.

Your POV: the fact that macros are not fully integrated with the rest of the language is a problem and we can not integrate them unless they would be fully hygienic, but if they would be fully hygienic and integrated then people would love them and switch to these fully hygienic “macros 2.0” thingie. Or we would develop nice rules for avoiding full hygiene 100% of time, which would avoid confusion and which people would understand.

My POV: without full hygiene attempt to “integrate fully” macroses with other things would create warts and the confusion (as my examples with #[macro_export] demonstrates) yet, most importantly, these same people who complain about confusing scoping rules of macro_rules! also implicitly very often rely on macros not being fully hygienic. So best case scenario would be if “macros 2.0” would just be abandoned and worst case scenatio — we would be stuck with two macro systems for the foreseable future. And people would continue to complain about macro_rules! — yet would use them and not “macros 2.0”.

Time will tell who is right, of course.

This point, and kind of this whole thread, has been mixing the scoping of macros themselves (macro namespacing) with the scoping/resolution of the items referred by the macro expansion (hygiene).

I've already tackled both, but mostly hygiene, in my previous post:


Now back to the OP question of how the macros are, themselves, scoped, and mainly to refer to

the following code compiles fine:

mod m {
    use crate::mac;
    
    cam!();
    mac!(); // defines `cam!` in this scope (`m::*`)
}

mod macros { // defines `mac` in this scope (`macros::*`)
    #[macro_export]
    macro_rules! mac {() => (

        #[macro_export]
        macro_rules! cam {
            () => {
                pub mod foo {
                    pub struct S;
                }
            }
        }
        pub use cam;

    )}
    pub use mac;
}

use m::foo::S;
Some history / context to explain why your snippet didn't work

The issue you had with your example had to do with "meta-macros", which are, on their own, a quite complex piece of machinery / non-obvious code to implement those in the compiler (macro expansion can define new items that change the meaning of item resolution, and this iterative process needs to be consistent else the compiler bails).

The actual issue in here is indeed the design of #[macro_export] macro_rules! macros. Those do define, among the programming system, a meta-programming tool that dependent crates may use. At the time the approach was quite consistent with itself (use #[macro_use] on modules defining local macros, use #[macro_use] extern crate … for crates defining #[macro_export]-ed macros). Macros were, at the time, pretty self-consistent, but totally alien / inconsistent with the rest of Rust items.

In an effort to both remove the "redundant" extern crate annotations, as well as to start unifying the macro namespacing with the namespacing of other items, with the 2018 edition, #[macro_export]-ed macros were namespaced: rather than doing

#[macro_use]
extern crate lazy_static;

lazy_static! { … }

one could finally do:

::lazy_static::lazy_static! { … }

which means that they could also depend on a crate bar with its own lazy_static! macro, and not have to deal with name collisions. Macros no longer being global / un-namespaced items has been a big improvement in that regard.

The issue, here, is that, imho, a mistake was done when implementing this, which becomes clear with the unfair advantage of hindsight: rather than appearing namespaced wherein the #[macro_export] occurred, these #[macro_export]ed macros always appeared as namespaced directly at the root of their crate:

  • //! dependency/src/lib.rs
    
    pub mod foo {
        #[macro_export]
        macro_rules! bar {() => ()}
    }
    
  • //! src/lib.rs
    ::dependency::bar!(); /* instead of:
    
    ::dependency::foo::bar!(); // */
    

At the time it might have made sense, since, inside a crate, macros were still not namespaced at all, and thus it would have been impossible for dependency to re-export bar at the root, should they have so wished.

But, this, in term lead to other issues. Consider:

//! dependency/src/lib.rs

#[macro_export]
macro_rules! foo {() => (
    bar! {}
)}

#[macro_export]
macro_rules! bar {() => ()}

Because of the lack of proper hygiene[1], if a dependent were to call ::dependency::foo!(), they'd stumble upon a 'bar!' not found error. So a #[macro_export(local_inner_macros)] hack was created, but since these hacks still suffered from the "single global namespace for all macros in the universe", in order to follow the new trend of namespacing, $crate:: was created, allowing the above to be written as:

  //! dependency/src/lib.rs

  #[macro_export]
  macro_rules! foo {() => (
-     bar! {}
+     $crate::bar! {}
  )}

  #[macro_export]
  macro_rules! bar {() => ()}

This solved the issue of cross-crate calls, but then we had the problem of a local call to foo! failing because:

  1. $crate:: wasn't really defined, although it could (and does) fallback to crate.
  2. crate::bar! wasn't defined, since, locally, macros were never namespaced.

In order to palliate this, it was made so #[macro_export] macros would be namespaced from within the crate defining it itself (first time this occurred), and, for the sake of consistency with $crate:: paths, they had to be namespace-moved / teleported to the root of the crate:

mod module {
    foo!(); // Error, no `foo!` in scope
    crate::foo!(); // OK

    #[macro_export]
    macro_rules! foo {() => ()}
}

Here is where we can see that having namespaced macros at the root of the crate wasn't ideal to begin with.

Enter meta-macros

Prelude: name-resolution loop

Since macros have always been able to define new items that thus affected name resolution, there is this back-and-forth between the meta-programming system (macro expansion) and the programming / compiler pass (in charge of name / path resolution).

Indeed, consider:

   // first pass: resolves `unstringify` and `module::fun`.
   use ::unstringify::unstringify;
   use self::module::*;

   mod module { pub fn fun() {} }

   fun(); // first pass: this seems to refer to `module::fun`
// ^^^ second pass: this actually refers to the function from macro expansion.
// +++-----------------------------------+
                                      // |
    unstringify!("fn fun() {}"); //  ----+
                                 // when this is expanded, it shadows
                                 // the previous `fun`

So far so good, but this now is problematic with

#[macro_export]

Indeed, those are able to affect the namespace of the root of the crate from within arbitrarily nested modules! Combine that with the above interactive loop, and you get a big mess for compiler authors.

Hence why it was decided that the only way to occupy the "root of the crate" was through an explicit #[macro_export], and not through a macro-expanded one / through one obtained from macro expansion. Hence that error about "name resolution is stuck", or the one about macro-expanded-macro-export-macro-referred-by-absolute-path. The solution is then to use, inside that crate, a name resolution path that does not involve the one at the root of the crate, such as:

mod macros {
    #[macro_export]
    macro_rules! m {() => ()}
    pub use m; // OK, "old/non-namespaced" path
}
macros::m!(); // OK

  1. This is the only time hygiene plays a role in the story of macro namespacing. ↩︎

8 Likes

That's nice, thanks. So if you add pub use in addition to #[macro_export] then you can do one more step.

But why this doesn't compile?

I just moved cam use to the other place.

And it's supposed to be there since it's exported as #[macro_export], no?

I think it's worth noting that there was some small momentum along allowing

pub macro_rules! m { ... }

(or pub(crate), or any pub(in path)) to define and mount a macro_rules macro exclusively and properly in the name resolution where it's defined (but still using mixed site span hygiene).

This was reverted due to some more issues, but I think would help continue to incrementally improve the situation for new code.

Yes, the compiler has to continue to support legacy macros and macro scoping in perpetuity, and legacy codebases will still have them, but that's no reason not to continue to make things better for the future. Plus, all of these new macro features are still in very early development, and the intent is that (just like editions) a crate should be able to upgrade to the newer, more principled way of doing things without impacting downstream users.

(And to that point: macro syntax is explicitly just a "it kinda works" placeholder such that full def site hygiene can be worked on. It can and will evolve and change when it comes time to design and RFC how that side of "macros 2.0" works, after the complicated, long-lasting, cross-cutting work is done to support the hygiene improvements to the compiler internals.)

3 Likes

It works if you do:

  mod n {
      use crate::cam;
  
      cam!(); // defines `mac` + `pub use mac;`
  }
  
  mod m {
-     use crate::mac;
+     use crate::n::mac;
      
      mac!();
  }

Feel free to look at the bottom of my "" section in my previous post to detail this: for historical reasons, Rust has to deal with the cruft of not being able to use crate::macro! for a #[macro_export]ed macro definition when it is macro-generated (and when inside that crate: external crates won't have this problem since there is no "feedback loop" as I mentioned):

mod n {
    use crate::cam; // OK we have an explicit `#[macro_export] macro_rules! cam`

    cam!(); // *expands* to `#[macro_export] macro_rules! mac`
}

mod m {
    use crate::mac; // Error, can't use such a path

Hence my workaround about defining such macros scoped to a non-root module, so that we can append a pub use … right afterwards, and be done with it. More generally, I listed workarounds in the canonical issue about this limitation:

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.