Name resolution question in macros

help
1

I just ran across this issue while using my first kind-of-useful macro (a helper for parsing AoC input) -- 6 years later, is this still the recommended way to approach this?

Specifically, I had naively placed the use anyhow within the macro itself, which worked until I decided to try out eyre and the compiler complained about not being able to find anyhow. I reviewed the reference on hygiene and made these initial changes. I found surprisingly few relevant SO results and tried to peek at the anyhow! and eyre! macro source, but I don't see external crates being referenced there.

Specifically, some of the issues I ran into (and didn't expect since the macro was working fine until I tried to change to eyre):

  • I initially placed use anyhow within the macro itself. This worked until I used the macro in a crate that didn't already depend on anyhow; it tries to resolve this dependency from within the new crate and can't find it.
  • I then tried place use anyhow in the crate containing the macro, in which case the macro can't find anyhow and the tests fail. (I still find this a little surprising, given that the crate depends on anyhow -- shouldn't the macro within the crate know about it too?)
    • Additionally, the compiler complained about an unused dependency; changing to pub use anyhow took care of that.
  • I then tried pub(crate) use anyhow; the macro still couldn't find it, so I went back to just plan pub.
  • I then tried $crate::anyhow in the macro, still couldn't find it.
  • Finally I realized that my macro was in a module, so pub use anyhow at the top level, then use $crate::files::anyhow in the macro
  • I tried going back to pub(crate) use anyhow at the top level, thinking that I didn't really want to reexport all of anyhow, but it still won't work that way (which makes sense, the consuming crate needs to know about anyhow if that's the resulting type).
  • I also had some issues with Context trait ambiguity between eyre and anyhow which required me to use the fully-qualified syntax.

After a few more tweaks and finding this thread, my resulting code follows the pattern:

pub use anyhow;

#[macro_export]
macro_rules! my_macro {
    ($path:expr, $ty:ty) => {{
        use ::std::example::stdlib::Dependency;
        use $crate::my::mod::name::anyhow;
        
        do_stuff_with_Dependency_and_anyhow();
    }}
}

I know the arch forum would throw a fit about a necrobump like this, but hopefully this will help another noob use external crates in their macro, because I had a hard time figuring this one out.

2

Can you, please, while experience is fresh in your mind, try to tell us why these listed solutions can make any sense? Like: any whatsoever?

Note, I'm not saying you are stupid. Far from it. Perhaps Rust is just not logical and have to be changed?

But… the main rule macros follow are: token are tokens and nothing except tokens exist.

Which, of course, mean that anyhow is just a token, too. Not variable or type or crate. Just a token.

If said token have no meaning in the target crate — it wouldn't be interpreted properly there.

Like: pub(crate) makes type anyhow visible to your crate but makes it impossible to import it from other crates… which, of course, means that target crate wouldn't be able to import it and thus the whole thing will fail.

What kind of mental model for how macros work have you had in mind when you tried these solution?

3

Ouch. Perhaps there are some language barrier issues here, but if the latter was not your intention, I think you might reconsider your delivery.

I had never considered this, or at least it had never sunk in. I've read TRPL, the little book of macros, parts of the rustonomicon... but I've never really written macros (I mentioned above this is my first that I've found any use for). I had thought that keywords like use were still keywords in a macro, but I guess they're just the literal tokens and essentially copied into the caller as u-s-e?

What kind of mental model for how macros work have you had in mind when you tried these solution?

I'm a hobbyist that tinkers with programming in my spare time; I've used python the longest, and I've never used a language with macros / templates, so python is probably where my mental model comes from. When you import a type / function / class in Python, the imported item seems to reference its parent scope without issues. I had expected the macro to be able to reference the environment in which it was defined.

Honestly I think the part that hung me up the longest was forgetting that I was in a module and not the root of the crate; I was very confused as to why use $crate::anyhow wasn't working, which put me on the wrong track.

4

Sorry about that. I just look on what you have tried and then try to imagine the set of rules which may govern the language where these attempt would make any sense. And fail.

You had some mental model of what is happening when you did these tries, right? How was it looking? How was it different from the model used in Rust? Perhaps that model is actually better (this is needed to ask “is this still the recommended way to approach this”, right?).

But I fail to imagine how what you was supposed to work. So… maybe I'm stupid? But then… what am I missing?

Now you are confusing me. Little book is, actually, very-very explicit: no, use is not copied as u-s-e. It's copied from input to output as one indivisible token: use.

  • Type of token: keyword.
  • Contents of the token: u-s-e.

Just like on picture from Wikipedia: text is parsed and turned into a syntax tree. Keywords exist in that representation. String literals exist. Numbers. Identifiers.

Tokens in — tokens out. Simple. Tokens, tokens, tokens. Macros don't know anything else.

But variables? Nope. Types? Absolutely not. Lifetimes? Oh, sure, there are lifetime tokens. But no lifetimes yet. Crates? What are you talking about: me know no any crates! Oh, you mean $crate ? Yeah, I know it. It's also a token, of course.

use is already a keyword (that is: keyword token), but it does not yet have any meaning! Just a stamp this is a keyword, beware.

Well… Python doesn't have macros just yet, but, of course, macros from PEP 638 are designed to behave just like macros from Rust: tokens, tokens, tokens, there are nothing but tokens in this world.

When you import a type / function / class in Python, the imported item seems to reference its parent scope without issues.

Yes, same with Rust. But that happens later, much later. When macros do their work scopes, variables, other, more complex, things… they don't exist yet. Tokens, tokens, tokens… nothing more.

It's very close to just search/replace work (except for hygienic macros). The simple model to follow: text separated in small pieces (tokens) and colored (like little books explains).

Macros are moving around these small sausages, but then, eventually, pass the whole thing to the compiler to interpret.

Scopes, variables, lifetimes and other such things are created from these small sausages much later. Little book is super explicit about that: at some point after the construction of the AST, but before the compiler begins constructing its semantic understanding of the program, it will expand all macros.

Semantic understanding of the program is, of course, time when sausages with some stamps on them stop being just sausages with some stamps and become something more complicated. Some would turn into variables, some into types… some would even bring entities from other crates… but all that would happen later.

Ah, got it. Well, I can understand the frustration. It's something many newcomers from many other languages where #include (or {$I, etc) works purely textually. But Python's module works similarly to Rust's mod, right? If you do import A and then inside of that module there are import B then variable in module B is accessed like A.B.V and not A.V…

5

The link is broken (extra symbol, it seems?)

6

No worries. I think your intentions are good. Thank you for this very thorough response, I'm sure you spent a nontrivial amount of time composing it, and it's very helpful.

I suppose it's mostly a model of how functions work. (Which I now understand is incorrect.)

Right, I just meant that it is copied as use with no other contextual information about what use means, not that it copies letter by letter. In other words that use may as well be asdf at this point.

This will be an interesting development, but it certainly doesn't reflect my familiarity with python over the last decade or so.

Right -- and I think therein was my confusion -- contrast this with your earlier point:

It wasn't different than the model used in rust -- just in the model used in a different context in rust.

In both Python and Rust I can have module or a function that references an external crate that was imported at the parent scope (hopefully that's the right terminology), and when I import that from a different crate it all sorts itself out without difficulty. As an end user, that works in Rust roughly like it works in Python.

Thanks again for the links to the links to sections of the little book; I've read it twice before, but it's been a while, and clearly there are concepts that I still don't grasp and will hopefully understand better as time goes on and I gain experience. Certainly ideas like Tokens, tokens, tokens… nothing more vs (from your link):

Rust has various kinds of tokens, such as:

  • Identifiers: foo , Bambous , self , we_can_dance , LaCaravane , …
  • Integers: 42 , 72u32 , 0_______0 , …
  • Keywords: _ , fn , self , match , yield , macro , …
  • Lifetimes: 'a , 'b , 'a_rare_long_lifetime_name , …
  • Strings: "" , "Leicester" , r##"venezuelan beaver"## , …
  • Symbols: [ , : , :: , -> , @ , <- , …

still has me a little confused. But clearly a crate is not a "kind of token" that is recognized or treated any differently.

Thank you again for the time, I think I understand things better now.

7

Well… I guess it was obvious to you, but it was definitely not obvious to me. Because traditional K&R C actually copies letter by letter. From the GCC documentation about how traditional C differs from ANSI/ISO C:

#define suffix(x) foo_/**/x
suffix(bar)
     → foo_bar

Ah. You just missed the forest for the trees when book started listed various kinds of tokens. The relevant part is little bit below:

The AST contains the structure of the entire program, though it is based on purely lexical information. For example, although the compiler may know that a particular expression is referring to a variable called " a ", at this stage, it has no way of knowing what " a " is, or even where it comes from.

It then goes on to explain how TokenTrees (which are what macros in Rust deal with) differ from both flat list of Tokens and fully-parsed ASTs.

Thus the important part (rust macros deal with tokens and tokens only and only care about lexical form but not semantic) ended up in the middle of quite large chapter.

8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.