Accumulate state across macro calls

#1

Is there some way to store values externally, in a macro call, so that over multiple invocations of the macro I could build up a collection of those values?

The context: I'm thinking about a Rust wrapper for a C++ API with virtual function callbacks, like Button in How to use C++ polymorphism in Rust

What's missing there is that in such classes, you don't usually implement all the callbacks. There may in fact be quite a boatload of callbacks, of which most are rarely used. The Rust wrapper can implement them all and let the library user override the default definitions (I think), but that's a fair amount of function calls and marshaling for virtual functions where perhaps no one will ever actually provide a non-default implementation. So if there were a convenient way to pick up which functions the user actually does implement, it might be worth some trouble.

like,

impl Button for CounterButton {
      #[buttonVfn(Click)]
      fn click(&mut self) {
      ...

(Which also takes care of the C++ API's camel case.)

I'm sure there's a means in Rust's intriguing macro system to pull an enum ButtonCallback::Click(click) out of that -- but I need to put it somewhere, so I think (?) I need to accumulate those data in a compile time collection, to, say, dump them out at the end of the impl block in an endButtonVfn!() or something. I tried redefining a macro, but Rust just saw that as an ambiguous set of macro definitions.

#2

Macro's are local i.e they operate on and have knowledge of only the pieces of syntax that they are applied to, so no that's not directly possible.

You can make it work with a hack though: create a state object and manually pass it to the macro calls, so have that the macro calls accumulate the state in there.

This does impose some limitations on how the macro can be used: unless you're using global state (which is a bad idea if you can avoid it), the macro can only be called inside a function, a method, or another macro.
Specifically, you can't call the macro at the item level i.e. the level at which most types, free fns etc are defined.

#3

Yeah, I can't see how a run time state object is going to be able to pick this up. A (procedural I think) macro can't emit a (declarative) macro definition, that would override a previous macro?

#4

Regarding the default impls of methods, I guess you are already aware that traits can do that, no need for a stateful macro. I have looked at the attached blog post, and I guess you are speaking of having the macro play some role in the generation of the glue for a manually laid out virtual function table, so as to cross the FFI layer with it and provide it to some C++ library, which offers an idiomatic C++ object wrapping the manually hand-rolled vtable.

And you'd like to optimize the virtual lookup part to only the methods that may be override, is that it? (This way you could provide default implementations that rely on static dispatch).

If that's the case, then there are several hacky ways to accumulate state over different macro calls:

  • the best one, but most difficult to achieve, is to use type-level properties with helper traits to encode the properties you are interested in. See ::frunk to see how far we can go with type level properties.

  • another one, depending on how far away the different invocations span across each other, is to wrap all the macro calls within an external macro call, that thus gets to witness / see all the inner invocations at once, within a single invocation, so the problem of state across invocations is no more:

    #[preprocessor_macro]  // Gets to see both.
impl Button for CounterButton {
    #[button....]  // <- first invocation (not truly a macro, just an annotaiton for the preprocessor)
    fn click (...) { ... }

    #[button....]  // <- second invocation (ditto)
    fn other ...
}

    Obviously this in your case won't work if you have multiple impl blocks spanned across multiple files.

  • The following option is just the previous one but on steroids: use a build.rs script to act as a preprocessor of all your codebase (macro_rules! have no power there, you will need to use things such as ::syn and recurse into the source files manually. Would require a ton of work, since I am not aware of tooling to help into that nested source file traversal).

  • Another option, especially for script generation, and which I have personally used to implement ::safer_ffi's header generation magic, is to use ::inventory to keep appending elements to some "global" (it achieves this by using "life before main" / ctors), and then have a helper test function that iterates over that generated collection to do its "build script logic". Not always doable, but for your use case of helper glue code / shim generation, it does seem like the most appropriate option.

  • A very hacky option, is to abuse the fact that procedural macros have currently access to the filesystem (although this is not documented anywhere, so it could break in a future Rust release (counter-counter point: I think too many people may already be relying on these hacks for it to be disabled by default)), so you can use that for persistence: the procedural macro {de,}serializes its state at each invocation using some file to hold it.

1 Like
#5

Given your use case, with a nightly / unstable feature, one can implement type-level sets (of a bounded number of elements), quite easily using #[marker] traits:

define_elements! {
    A, B, C,
}

enum Set {}

insert!(A in Set);
insert!(B in Set);
insert!(A in Set); // the nightly feature is required to prevent redundant inserts from causing a compile error (redundant impls).

fn main ()
{
    assert_eq!(contains!(A in Set), true);
    assert_eq!(contains!(B in Set), true);
    assert_eq!(contains!(C in Set), false);
}
#6

My understanding about the danger here is that the compiler makes no guarantee about the order procedural macros will be invoked, and future incremental compilation changes might lead to the macro not being invoked at all for partial compilation runs, when a previous output is available.

If you try to store state in the file system, it will be hard (or impossible) to keep it synced properly with the compilation cycle, and some future version of the compler might decide to expand your summary macro before the macro calls it’s supposed to be summarizing.

1 Like
#7

Thanks! Lots to think about there. I note that "many people" are squirreling state from macros into disk files ... hm. Not that it's where I'd want to go.

I will need to look harder at the type-level stuff - I'm about one week into Rust, so not much sinks in at first glance - but likely I'll make a note to come back to it and see how it's going.

And meanwhile try the macro outside the block - it seems to me perfectly reasonable to require all callbacks to be defined in the same block - like, why didn't I think of this? The "not really a macro" annotation is interesting - I see it has to have been defined, but how it's defined apparently matters not.

#8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.