Passing a data structure between two proc macros?

help
1

I have a proc macro pattern where one upstream producer proc macro needs to take input and compute intermediate data for other downstream consumer proc macros to later use. The producer has to be a proc macro rather than an external configuration because part of its process involves checking #[cfg(...)] conditions in the calling crate (see here for an explanation).

My plan for doing this is for the producer macro to output several macro_rules! constructions of child consumer macros, like so:

// This is output from the producer_macro! proc macro:
macro_rules! consumer_macro_1 {
    ($(args:tt)*) => __handle_consumer_macro_1(/* DATA GOES HERE */, $($args)*)
}

and then __handle_consumer_macro_1 is a proc macro that takes in the generated data (generated by producer_macro), and uses that data to produce output from the arguments.

Currently my best plan is to serialize the intermediate data and include it as a base64 string literal. So it would be something along the lines of:

// This is output from the producer_macro! proc macro:
macro_rules! consumer_macro_1 {
    ($(args:tt)*) => __handle_consumer_macro_1("SGVsbG8gd29ybGQh", $($args)*)
}

but I'm curious if there's a better way. I could also just output the data structure I'm trying to pass and re-parse it, but my hunch is that that would be slower than passing a serialized form.

Has anyone else been down this road before? Any handy solutions to this problem?

2

Parsing is the fastest part of the compilation process. What makes Rust compilation slow is type checking and IR optimization (and sometimes linking). You don't realistically need to worry about base-64 vs TokenStream.

1 Like
3

Is there a straightforward way to output a data structure as a TokenStream that's trivially parsable again? I've actually found it easier thus far to serialize into base64 using speedy and the base64 crate (only about 2-4 lines of code).

4

I mean, you can just literally quote!() literal initialization syntax; I had the impression that was what you were already doing.

5

It's tricker in this case. I have to pass something in as a proc macro argument that then gets turned into an instance of a data structure.

EDIT: To elaborate a bit on the original post above, the process is like this:

I have three crates: my my_proc_macros crate, the user's upstream crate, and the user's downstream crate. In the user's upstream crate, they declare a data processing macro like this:

use my_proc_macros::preprocess_data;

preprocess_data! {
    // Input goes here
}

This process the data and then spawns child macros (still in the upstream crate) to query that data. Those child macros have the data baked-in to their arguments as above:

// This is generated by the preprocess_data! invoke:
macro_rules! query_data{
    ($(args:tt)*) => ::my_proc_macros::do_query_data!("SGVsbG8gd29ybGQh", $($args)*)
}

Where the "SGVsbG8gd29ybGQh" string literal right now is the base64 encoding of the post-processed data. The my_proc_macros::do_query_data proc macro takes in, as arguments, both the string literal post-processed data blob, along with the arguments for the actual query the user wants to perform, and produces a result. This way, the user in the downstream crate can call query_data! and that downstream macro can access the baked-in encoding of the preprocessed data.

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.