Generating JSON based on trait


#1

I’m contributing in a library which uses proc macro to generate JSON and some structs based on trait signature.

The problem is that in this case this library adds serde_json as a runtime dependency while it’s only used when building the library. It makes sense to move it to the build.rs and make serde dev-dependency. But here is a catch: I didn’t find any way to convert source file into token stream. syn crate only works if you have TokenStream, but in my case I don’t have.

How could it be done? Am I missing something or there is no way to do it with today architecture? It looks quite common that your attribute may use some dependencies that aren’t actually runtime ones.

If you wonder why I’m worried: there is a known bug (feature?) in the cargo that it cannot resolve one library linked as std and no_std (I have seen 6 or 7 issues about it). And this is exactly what I have. So I want to move one into dev-deps to resolve the conflict.


#2

TokenStream implements the standard library’s FromStr.

let tokens: TokenStream = string.parse()?;

It also implements Synom: (edit: this is irrelevant because syn::parse requires you to already have a TokenStream)


#3

Thank you, gonna try it tomorrow.

P.S. Is it known practice? Because I have never seen it before in other crates… It seems a ubiquitous task (in most cases codegen require much more deps than resulting code), but everybody just puts everything in proc_macro_attribute fn


#4

Anyway, is there the better way to do it? Because currently I only see following:

  1. find all occurences of custom_attribute in the code
  2. extract somehow the whole method decorated by it
  3. parse()
  4. use existing code that was creating runtime dependencies.

Isn’t it too complicated? May I be missing something?


#5

It’s something more like this:

  • Read the entire file to a string.

  • Parse the entire string into a TokenStream.

  • Use syn::parse on the entire TokenStream to obtain a File. This gives you the full AST.

  • Find Items that have the attribute and do something with them. This is most easily done by creating one of the following:

    • a Visit is basically a glorified FnMut(&T)
    • a VisitMut is basically a glorified FnMut(&mut T)
    • a Fold is basically a glorified FnMut(T) -> T

    Whichever one you implement, you should only need to override a small number of functions on the trait. (based on your post, you probably want to override either foo_item or foo_item_fn. (But don’t override both, as that would be redundant!))

  • New AST nodes can be generated with parse_quote!. You can also use quote! directly to turn an arbitrary AST node back into a TokenStream if necessary.


#6

Yep, I see that there is a visitor. Thank you.

Maybe better alternative is write custom cargo command to build this JSON?.. I’m not quite sure. Generating it in attribute is awful, but the JSON shouldn’t be generated if build failed. But if I write build.rs which runs before build it JSON could be generated while build failed. Need some investigation here too.