Proc Macro: Create Static Structs

1

Hi Rustaceans!

I want to create a proc macro that

  • reads a file
  • instantiates a struct
  • returns a token stream of this instance

The instantiated struct must be available at runtime, but I'm not sure how to derive the corresponding token stream.

More details about the use case:

The file contains definitions about entities, i.e. field names and types, in a domain-specific language. It can be parsed to an equivalent Rust struct which holds the needed metadata to derive the individual entity struct definitions. At runtime, I'm not only interested in the individual entity struct definitions but also in the metadata struct itself.

Is there a way to return a token stream for the individual entity struct definitions and the static metadata struct which was already built inside the proc macro?

In pseudo code:

let file_content = fs.read_to_string(...)
let metadata = Metadata::parse(file_content)
let struct_definitions = create_tokenstream(metadata)
return "metadata and struct definitions"

Maybe there is also a better approach, my only requirement is that the file is parsed during compile time to derive the entity structs. At runtime I also need the metadata.

I appreciate any help, thank you and best regards,
David

2

I'm having a little trouble understanding your problem, mainly what metadata means in your context:

Could you elaborate this a little?

This is also greatly confusing to me. What is an instantiated struct to you? Do you mean a constant? Also, what is a static metadata struct?

If you have two individual token streams, you can combine them into one and return it from your procedural macro, if that is the only problem you are facing. It's quite easy when you use proc_macro2::TokenStream with the quote::quote macro, see here.

3

Thanks for the reply!

The metadata struct is something like this:

struct Metadata {
  entities: HashMap<String, Entity>,
}

struct Entity {
  name: String,
  fields: HashMap<String, Field>,
}

struct Field {
  name: String,
  field_type: FieldType,
}

enum FieldType {
  String,
  Integer,
}

Now I want to create a proc macro which reads a file and constructs the Metadata struct, i.e.

let metadata = Metadata { entities: vec![...] };

where ... is based on the file content. I already implemented this (Metadata::parse(file_content)).
I want to return this metadata struct from my proc macro, in order for users of the proc macro to have access to it. I need so somehow transform it to a tokenstream, but I don't know how.

In addition to that I want to return struct definitions based on the information in the metadata struct, e.g.

struct MyEntity {
  my_name: String,
}

coming from

Entity {
  name: "MyEntity",
  fields: HashMap {
    my_name: Field {
      name: "my_name",
      field_type: FieldType(String)
    }
  }
}
4

There is

But unfortunately it only outputs source text, not a token stream. You could parse its output with syn, or write your own conversion from struct to the tokens of a struct literal.

5

Thanks, that's a really helpful reply!

How about something like this? Pseudocode using quote to construct token stream:

let file_content = fs.read_to_string(...)
let metadata = Metadata::parse(file_content)
let struct_definitions = create_tokenstream(metadata)

let entities = metadata.entities.iter();

quote! {
    struct Metadata {
        entities: std::collections::HashMap<String, Entity>,
    }

    impl Metadata {
        fn new() -> Metadata {
            let mut entities =  std::collections::HashMap::new();
            
            #( 
                entities.insert(#entities.0, #entities.1);
            )*
            
            Metadata { entities }
        }
    }
    
    // ... here you need to declare Entity and other types. 
    //
    // Also a good idea is to wrap this whole stream in a module
    
    #struct_definintions
}

Not 100% sure about if interpolation of #entities.0 is going to work, probably not. Then you'd have to create two iterators, one with the keys and one with the values (e.g. with unzip()).

I expect the program that calls your procedural macro to look something like this then:

use your_crate::your_macro;

// create the type definitions 
your_macro!();

fn main() {
    // here we get access to the metadata
    let metadata = Metadata::new();
}
6

Thanks so much, that would work! The only downside is that I have to manually construct the new function which becomes cumbersome if the metadata struct increases in complexity. But I guess there's no good way around it!

Again, thanks so much for this valuable response!

1 Like
7

Right now I don't think there is. Once constant evaluation becomes more mature, it may be possible to construct HashMaps in a const context. Then you could add a constant to your type, instead of a new method:

struct Metadata {
  entities: std::collections::HashMap<String, Entity>,
}
  
impl Metadata {
  const INSTANCE: Self = {
      let mut entities =  std::collections::HashMap::new();
      
      #( 
          entities.insert(#entities.0, #entities.1);
      )*
      
      Metadata { entities }
  };
}

use your_crate::your_macro;

// create the type definitions 
your_macro!();

fn main() {
    // here we get access to the metadata
    let metadata = Metadata::INSTANCE;
}

Edit: I may have misinterpreted your reply. Yes, constructing token streams can become very complex very fast, but tools like quote make it much easier.

1 Like
8

databake by the icu4x team also seems relevant w.r.t. "'serialize' to Rust code."

9

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.