How to use derive macro to parse the meta information of a struct

I am using struct to build the schema of the database, so I need to meta information of struct.

Suppose my schema struct is defined like below:

#[derive(Debug,Clone,Default,Serialize,Deserialize)]
pub struct User {

  #[field_attr( unique = true, default = "", index=["@hash", "@count"] )]
  pub name: String,

  #[field_attr( unique = true, default = "" )]
  pub username: String,

  pub description: Option<String>,
  pub age: u32
}

I want to parse it into the following struct:

pub struct Schema<T>{
  name: String,
  origin: T,            // the struct to be parse, like the User struct above.
  fields: Vec<Field>,   // the struct's fields
}

// the struct's fields info
pub struct Field {
    field_name: String,
    field_type: String,
    field_attrs: FieldAttribute
}

// the field's attr info
pub struct FieldAttribute {
    unique: bool,
    default: String,
    index: Vec<String>   // Index of the database
}

Suppose I use json to express, its result should be as follows:

{
    name: "User",
    fields:[
        {
            field_name : "name",
            field_type : "String",
            field_attrs : { unique : true, default : "", index: ["@hash", "@count"] }
        },
        {
            field_name : "username",
            field_type : "String",
            field_attrs : { unique : false, default : "" }
        },
        {
            field_name : "description",
            field_type : "String",
            field_attrs : null
        },
        ...
    ],
    ...
}

I know it can be achieved using derive macro, but I don’t know how to write it.

I have only recently begun to learn derive macro and have not fully mastered it yet. It is difficult to find useful tutorials about derive macro on the Internet.

Hope to get help.

Usually, the syn crate is used for parsing input to derive macros into an AST. Then you can process the resulting AST manually, or use higher-level creates, like darling.

@H2CO3 Thanks. Is there any example code? I don't know how to write, this has been bothering me.

Have a look at the examples in the darling repository. As far as I can see, all four use attributes (both struct and field) so that should give you somewhere to start.

@Michael-F-Bryan
Thanks, although this does not meet my expectations.

I need to write a custom derive macro to achieve it. I wrote a beginning, but I don’t know how to write specific implementation details.

The code i started to write:

#[proc_macro_derive(ParseStruct)]
pub fn parse_struct_derive(input: TokenStream) -> TokenStream {
    let ast = syn::parse(input).unwrap();
    let name = &ast.ident;
    
    let gen = quote! {
        impl #name {
            fn parsed_schema()-> Schema { 
               // return the parsed schema     
               //...           
            }
        }
    };
    gen.into()
}

Expected result:

use parse_struct::ParseStruct;

#[derive(ParseStruct)]
struct User{
    #[field_attr( unique = true, default = "", index=["@hash", "@count"] )]
    pub name: String
}

fn main() {
    let schema = User::parsed_schema();
    println!("{:#?}",schema);
}

I don't know how to implement it. :expressionless:

If you're interested in spending some more time learning these APIs, https://github.com/dtolnay/proc-macro-workshop is invaluable. It's a set of tutorials/workshops on writing proc macros which start out simple, and get progressively more complex. By the end, I think you should be able to write something like what you're wanting to do.

2 Likes

@daboross Thanks, this is very helpful to me, I will take the time to learn. But now I urgently need an answer, otherwise my work cannot continue.

That's reasonable!

I don't think what you're doing is going to be simple enough to be answered in a single post here, but hopefully we can provide some of the pieces.

Also unfortunately, there's not a lot of "documentation-like" documentation for proc macros that I can point to. Most of the details are really only documented through tutorials, like in the rust book or that workshops repo. https://doc.rust-lang.org/stable/book/ch19-06-macros.html#procedural-macros-for-generating-code-from-attributes is shorter than the workshops, and you can just read it, so maybe it'd be worth a read? It's a canonical source for at least some of this information. It's also a beginner source, though, so maybe not.


As for actually writing what you want to write, I think you're close-ish, but you'll need a lot more machinery.

First, to accept field_attr, you'll need to declare the name of the attribute in proc_macro_derive, like so:

#[proc_macro_derive(ParseStruct, attributes(field_attr)]
...

Next, you'll want to have some way to walk the AST, and then some way to generate and combine data from that. Walking the AST is relatively simple, it just requires looking into syn rustdocs, finding the data structure you have, and looking at its fields. Most things have iterators, and you can just iterate through the fields and whatnot.

An easier alternative to that is to use a helper crate. I'd recommend synstructure for easily walking through fields and collecting stuff. It gets rid of most or all of the boilerplate, so you just write how to turn a single item

  #[field_attr( unique = true, default = "", index=["@hash", "@count"] )]
  pub name: String,

into

Field { field_name: "name".to_owned(), field_type: "String".to_owned(), ... }

Lastly, you'll probably want to be constructing your result piece-by-piece, rather than trying to do it all at once. This applies whether you're using synstructure or manual iteration. I think the easiest way to do this is using ToTokens::to_tokens to append to an existing stream. For example,

let mut fields = proc_macro2::TokenStream::new();

(quote! { Field { field_name: #name, ... }, }).to_tokens(fields);
(quote! { Field { field_name: #name, ... }, }).to_tokens(fields);
let constructor = quote! {
    Schema {
        name: #schema_name,
        origin: self,
        fields: vec![ #fields ],
    }
};

Took me forever to figure out how to do that nicely! There's probably a better way, but that should work for everything you need to be constructing.

Hopefully this gives you some tools to use! Let me know if there's anything else in particular that seems hard to figure out?

I strugle myself to find some usable code examples for a very similar use-case.
I built this with a lot of trial-error.
Look at the code if that can help you:


It is used in

@daboross
Thanks, this gives me a better understanding of derive macro.
But since I have only recently begun to learn derive macro, my understanding of derive macro is very fragmented, and I can only write part of the code. For example, ToTokens you mentioned, I am very new to it.

I uploaded my code to github, here is my code repository:
parse-struct

I wrote part of the code, but I don't know how to continue writing.

Your code is very helpful, I am ready to try it, thanks.

@LucianoBestia
Although I did something similar to yours, there are still differences, and I encountered some problems.
The structure I need to parse is different from yours:

#[derive(Debug,Clone, Serialize, Deserialize)]
pub struct Schema { 
  pub name: String,
  pub fields: Vec<Field>
}

#[derive(Debug,Clone, Serialize, Deserialize)]
pub struct Field {
  pub field_name: String,
  pub field_type: String,
  pub unique: bool,  
  pub default: Option<String>,
  pub index: Option<String>
}

So I don’t know how to reuse your code.

My logic only needs to implement the parse_schema method, which returns the parsed Schema.

extern crate proc_macro;
use crate::proc_macro::TokenStream;
use quote::quote;
use syn::{
    Lit,
    Meta,
    Data,
    Fields,
    FieldsNamed,
    DataStruct,
    DeriveInput,
    parse_macro_input
};

#[proc_macro_derive(ParseStruct, attributes(field_attr))]
pub fn parse_struct_derive(input: TokenStream) -> TokenStream {
    let ast = parse_macro_input!(input as DeriveInput);
    let name = &ast.ident;

    let _fields = if let Data::Struct(DataStruct {
        fields: Fields::Named(FieldsNamed { ref named, .. }),
        ..
    }) = ast.data { named } else {
        unimplemented!();
    };

    let builder_fields = _fields.iter().map(|f| {
        let field_name = &f.ident;
        let field_type = &f.ty;
        let mut unique: bool = false;
        let mut default: Option<String> = None;
        let mut index: Option<String> = None;

        let attribute = get_field_attr(&f);
        match attribute.parse_meta() {
            Ok(meta)=>{
                if let Meta::NameValue(m) = meta {
                    if m.path.is_ident("unique") {
                        if let Lit::Bool(LitBool) = &m.lit {
                            unique = LitBool.value;
                        }                    
                    }
                    if m.path.is_ident("default") {
                        if let Lit::Str(LitStr) = &m.lit {
                            default = Some(LitStr.value());
                        }            
                    }
                    if m.path.is_ident("index") {
                        if let Lit::Str(LitStr) = &m.lit {
                            index = Some(LitStr.value());
                        }            
                    }            
                }
                quote! {
                    ::parse_struct::Field {
                        field_name: #field_name,
                        field_type: #field_type,
                        unique: #unique,
                        default: #default,
                        index: #index
                    }
                }
            },
            Err(err)=> quote! { err }
        }
    });
    
    let empty = Vec::new();
    let gen = quote! {
        impl ParseStruct for #name {
            fn parse_schema() -> ::parse_struct::Schema { 
                ::parse_struct::Schema {
                    name: #name,
                    fields: #builder_fields   // get an error
                }
            }
        }
    };
    gen.into()
}

fn get_field_attr(field: &syn::Field) -> &syn::Attribute {
    let attrs: Vec<&syn::Attribute> = field
        .attrs
        .iter()
        .filter(|at| at.path.is_ident("field_attr"))
        .collect();
    attrs[0]
}

I get an error:

the trait `quote::to_tokens::ToTokens` is not implemented for `std::iter::Map<syn::punctuated::Iter<'_, syn::data::Field>, [closure@parse_struct_derive/src/lib.rs:27:45: 66:6]>

I don't understand why this is, I hope you can help me.

My code on github: parse-struct .

This feature prevents me from continuing to work, it is too difficult, I still need to learn more :sob:

The end result you're trying to construct is a token stream, a series of tokens like "if", "x" , "let" and so on. The error is saying that quote doesn't know how to turn your iterator into a list like that, basically. See Repetition; you want to do #(#builder_fields,)* to insert each field separated by commas. Of course, what you actually want is a vec and not just a series of fields so do vec![#(#builder_fields,)*].

Another reoccurring issue in the macro is that you're using tokens where you want strings. For example, name: #name will output name: User which isn't valid syntax. You want the result to be name: String::from("User"), or something like that. I'm not sure if there's some shortcut for doing so, but you could do

let name_string = name.to_string();
...
name: String::from(#name_string),

Same applies to the field name and type when you're generating Fields. Unfortunately they don't seem to have a to_string method, so I did it like this: let name_string = quote::quote! (#field_name).to_string();. Might be that there's an easier way but it works.

The conversion of the Attribte to Meta also fails. It seems that index=["@hash", "@count"] is not valid syntax for parsing into Meta, so you'll have to express it in some other way.

Inserting an Option into a quote also doesn't do what you're looking for. A Some(string) will be inserted as string and a None will be inserted as (). Again I'm not sure what the best way to tackle this is, but using quote::quote!(None) and quote::quote!(Some(lit_str.value())); seems to work.

Edit: Debugging proc macros can be pretty tricky. If you're not using cargo expand you should definitely give it a try. Additionally, eprintln!("some log message") messages will be printed in the compile process and can be used to help out in the similar way as regular debug prints.

I think you've created an iterator over token streams, and that itself isn't a token stream. You'll want to collect the iterator into a token stream first - with something like

    let builder_fields = _fields.iter().map(|f| {
        ...
    }).collect::<TokenStream>();

Thanks, this solved my error.

@Heliozoa Thanks, you told me a lot of useful knowledge.

The result of builder_fields seems to be a vec, I just need to assign it to the fields field.

::parse_struct::Schema {
    name: #name,
    fields: #builder_fields
 }

Yeah. So I plan to change to index="@hash, @count".

I didn’t use quote, but directly assigned it to a variable defined in advance.

if let Lit::Str(lit_str) = &m.lit {
  default = Some(lit_str.value());
}     

I don’t know if this is wrong, but it didn’t report an error.

Finally. I made changes to the code based on the above suggestions:

extern crate proc_macro;
use crate::proc_macro::TokenStream;
use proc_macro2::TokenStream as TokenStream2;
use quote::quote;
use syn::{
    Lit,
    Meta,
    Data,
    Fields,
    FieldsNamed,
    DataStruct,
    DeriveInput,
    parse_macro_input
};

#[proc_macro_derive(ParseStruct, attributes(field_attr))]
pub fn parse_struct_derive(input: TokenStream) -> TokenStream {
    let ast = parse_macro_input!(input as DeriveInput);
    let name = &ast.ident;

    let _fields = if let Data::Struct(DataStruct {
        fields: Fields::Named(FieldsNamed { ref named, .. }),
        ..
    }) = ast.data { named } else {
        unimplemented!();
    };

    let builder_fields = _fields.iter().map(|f| {
        let field_name = &f.ident;
        let field_type = &f.ty;
        let mut unique: bool = false;
        let mut default: Option<String> = None;
        let mut index: Option<String> = None;

        let attribute = get_field_attr(&f);
        match attribute.parse_meta() {
            Ok(meta)=>{
                if let Meta::NameValue(m) = meta {
                    if m.path.is_ident("unique") {
                        if let Lit::Bool(lit_bool) = &m.lit {
                            unique = lit_bool.value;
                        }                    
                    }
                    if m.path.is_ident("default") {
                        if let Lit::Str(lit_str) = &m.lit {
                            default = Some(lit_str.value());
                        }            
                    }
                    if m.path.is_ident("index") {
                        if let Lit::Str(lit_str) = &m.lit {
                            index = Some(lit_str.value());
                        }            
                    }            
                }
                quote! {
                    ::parse_struct::Field {
                        field_name: #field_name,
                        field_type: #field_type,
                        unique: #unique,
                        default: #default,
                        index: #index
                    }
                }
            },
            Err(err)=> {
                quote! {
                    ::parse_struct::Field {
                        field_name: #field_name,
                        field_type: #field_type,
                        unique: #unique,
                        default: #default,
                        index: #index
                    }
                }
            }
        }
    }).collect::<TokenStream2>();
    
    let gen = quote! {
        impl ParseStruct for #name {
            fn parse_schema() -> ::parse_struct::Schema { 
                ::parse_struct::Schema {
                    name: #name,
                    fields: #builder_fields
                }
            }
        }
    };
    gen.into()
}

fn get_field_attr(field: &syn::Field) -> &syn::Attribute {
    let attrs: Vec<&syn::Attribute> = field
        .attrs
        .iter()
        .filter(|at| at.path.is_ident("field_attr"))
        .collect();
    attrs[0]
}

Everything works fine here, no more errors reported.

But when I use ParseStruct I got an exception:

use parse_struct::ParseStruct;

#[derive(Debug,ParseStruct)]   //  The ParseStruct throws an exception
struct User{
    #[field_attr( unique = true, default = "", index="@hash @count")]
    pub name: String,
    pub age: u32
}

fn main() {
    let schema = User::parse_schema();
    println!("{:#?}",schema);
}

The ParseStruct throws an exception:

proc-macro derive panicked

help: message: index out of bounds: the len is 0 but the index is 0

I can't see where the mistake is. I guess there may be a problem with derive macro, although it does not report an error.

Help me, I have updated the code: parse-struct.

The error is from attrs[0]. Your code assumes that every field has a field_attr property and panicks when it tries to parse the age field because it has none. You'll need to handle fields without attributes, add them to age or remove the field. Doing so will cause a lot more errors, but they can all be fixed with what I said in the previous post.

Based on your suggestion, should I do this?

    let builder_fields = _fields.iter().map(|f| {
        let _name = &f.ident;
        let _type = &f.ty;
        let mut field_name = quote! { #_name };
        let mut field_type = quote! { #_type };
        let mut unique = quote! { false };
        let mut default = quote! { None };
        let mut index = quote! { None };

        let attribute = get_field_attr(&f);
        if !attribute.is_empty() {
              let field_attr = attribute[0];
              // some code to change unique/default/index
        }
        
        quote! {
            ::parse_struct::Field {
                field_name: #field_name,
                field_type: #field_type,
                unique: #unique,
                default: #default,
                index: #index
            }
        }        
    }).collect::<TokenStream2>();

But i get some errors:

expected one of `,`, `.`, `?`, `}`, or an operator, found `::`
proc-macro derive produced unparseable tokens

I get a different error, did you change something else in your code? You should try cargo expand to see what the macro is producing and look for invalid syntax there to see where the issue might be (install with cargo install cargo-expand if you don't have it yet).

Oh. I did not submit the above code, I will submit it now.

@Heliozoa I have submitted.
I haven’t used cargo-expand, I’m trying to use it now.