Is it possible to match part of a syntax tree using `syn` and `quote`?

Is it possible to match part of a syntax tree using syn and quote? In a declare macro, I'd like to match and identify elements whose type is Vec. At the moment, I have unwieldy piece of code that accomplishes this, poorly, with

let v = if let Type::Path(ref p) = field.ty {                                    
    p.path                                                                       
        .segments                                                                
        .iter()                                                                  
        .next()                                                                  
        .unwrap()                                                                
        .ident                                                                   
        .to_string() == "Vec"                                                    
    } else {                                                                     
        false                                                                    
    };

Here, field has type syn::Field and the code returns true if the type has the form Vec <T> and false otherwise. This seems much more complicated than it should be. Rather, I'd like to have a piece of code that looks something like

let v = if let quote!(Vec<#type>) = field.ty {
    true
} else {
    false
}

As far as I know, the quote! macro can not be used in a pattern matching position, so this doesn't work. Is there either a different or similar method to accomplishing this kind of match compactly?

2 Likes

First: Ident can be compared to &str directly; no need to go through to_string.

Second: proc macros don't have access to type information. Any attempt to determine type information will fail in some edge case, and often in common cases. Wherever possible, let the type system do type system things, via providing runtime helper code, traits, and auto(de)ref specialization as required.

Third: your check doesn't even succeed in checking for use of the type named Vec in local scope, as it will also accept a path such as Vec::haha::your::macro::broke, as the first segment is Vec, or even ::Vec::macro::punisher if you had a crate named Vec.

Basically the only path checking you can somewhat reliably do is path.is_ident("Ident"), anything else will run into horrible edge cases super quickly. You'd think checking if a type is a known type could be simple, but due to macros' purely syntactic nature, it's full of problems and edge cases that the code represents here. Providing a simple "solution" would give the impression of being more complete than it actually is.

The closest to what you're asking for is syn::quote_syn!, which works like quote! but produces a syn type (at runtime) rather than a TokenStream. This still can't be used as a pattern or include undefined placeholders, though.

4 Likes

I very much agree that the situation isn't ideal and difficult to deal with. Though, I appreciate the info on how difficult this really is. That said, is there any reliable way to determine if we have a Vec? The code

let v = if let Type::Path(ref p) = field.ty {                                    
    p.path.is_ident("Vec")                                                       
} else {                                                                         
    false                                                                            
};

doesn't match properly because is_ident only works with path segments of length 1 and Vec will always be longer than that.

I'd be curious as to why you need to know a concrete type. It sounds very fishy. If the macro relies on the property that e.g. the field is a list-like collection, then just make a custom trait and generate code that works against that trait. Then if the user supplies a type that doesn't implement the trait, you'll get a proper, reliable type error.

1 Like

I would like to create a declare macro that allows some elements of a struct to be converted to a Vec and for that resulting Vec to be converted back into the struct. Essentially, I'd like to be able to write code such as

let s = MyStruct{ x : vec![1.2,2.3,3.4], y : 4.5, z : "I like pie" };
let v : Vec <_> = s.clone().to_vec().iter().map(|x|2.*x).collect();                    
let s = s.from_vec(v); 

and then have the result be the equivalent as

let t = MyStruct{ x : vec![2.4,4.6,6.8], y : 9.0, z : "I like pie" };

Certainly, not all elements can be converted and are compatible. That's fine. In addition, I don't need a perfect macro, but simply a useful one. While I believe you're correct that a smart use of traits can allow much of this to be done, I continue to have difficulty determining a way to accomplish what I need. For example, in the construction of the function to_vec we need to accumulate a vector from the marked elements. In the element is a Vec we can grab the elements using append. If the element is some kind of scalar, we can grab the elements using push. Is there a way to accomplish this without the macro identifying the type of the element?

So, in this case, how would you encode the string "I like pie" so that when it goes through map(|x| 2 * x), it doesn't change?

Because apart from this twist, it's possible to just implement a trait for supported types (e.g. integers, floats, and Vec for starters), which, in a way appropriate for each type, appends to a Vec, e.g. uses extend() for Vec-typed fields, push for integers or floats, etc. Then your macro only needs to generate code that creates a vector, goes through each field and calls the trait's method to append to this vector, then returns the vector, e.g.:

trait AppendToVec {
    fn append_to_vec(&self, vec: &mut Vec<f32>);
}

impl AppendToVec for f32 {
    fn append_to_vec(&self, vec: &mut Vec<f32>) {
        vec.push(*self);
    }
}

impl AppendToVec for Vec<f32> {
    fn append_to_vec(&self, vec: &mut Vec<f32>) {
        vec.extend(self);
    }
}

// And the proc-macro needs to generate code like this:
fn to_vec(&self) -> Vec<f32> {
    let mut vec = Vec::new();
    #(#field.append_to_vec(&mut vec);)*
    vec
}

This is not a particularly elegant or generic implementation, but you get the idea: it can be made into one once you get from the example how the pieces fit together.

1 Like

Do you mean ::syn::parse_quote! ?

1 Like

Whoops, yes I did. I just wrote that comment on my phone without checking.

1 Like

Alright, so as a follow-up, I can make the code work, but it feels more difficult than it needs to be. In order to deal with the issue of recognizing specific types, I just force the user to specify the type by hand through an attribute. Is there a more elegant way to make this work? The macro is specified as:

// External dependencies
extern crate proc_macro;
use crate::proc_macro::TokenStream;
use quote::quote;
use syn::{
    parse::{Parse, ParseStream},
    parse_macro_input,
    punctuated::Punctuated,
    Data, DeriveInput, Fields,
    Lit::Str,
    MetaNameValue, Token, Type,
};

// Parses options of the form (option="value",)*
struct Options(Punctuated<MetaNameValue, Token![,]>);
impl Parse for Options {
    fn parse(input: ParseStream) -> syn::Result<Self> {
        let content;
        syn::parenthesized!(content in input);
        let param: Punctuated<MetaNameValue, Token![,]> =
            content.parse_terminated(MetaNameValue::parse)?;
        Ok(Options(param))
    }
}

// Define a derive macro
#[proc_macro_derive(Vectorize, attributes(vectorize))]
pub fn vectorize_macro_derive(input: TokenStream) -> TokenStream {
    // Parse the target of the declare macro
    let ast = parse_macro_input!(input as DeriveInput);

    // Grab the name of the struct
    let name = &ast.ident;

    // Look for the vectorize attribute that contains our options
    let attr = ast
        .attrs
        .iter()
        .filter(|a| a.path.is_ident("vectorize"))
        .nth(0)
        .expect("vectorize attribute required for deriving Vectorize");

    // Parse our options
    let options: Options = syn::parse2(attr.tokens.clone()).expect(
        "vectorize attribute must be in the form \
         \"vectorize((option = \"value\",)*)\"",
    );

    // List of valid options
    let valid = vec!["base"];

    // Throw an error if we have any invalid options
    options.0.iter().for_each(|option| {
        // Requires the path to match a string in valid
        if !valid.iter().any(|&v| option.path.is_ident(v)) {
            // At this point we have a problem, so turn the path into something
            // that can be displayed to the user
            let path = option.path.clone();
            let path = quote!(#path);
            panic!("{} is not a valid option for attribute vectorize", path);
        }
    });

    // Extract the base type
    let base_type = options
        .0
        .iter()
        .filter(|option| option.path.is_ident("base"))
        .nth(0)
        .unwrap()
        .lit
        .clone();
    let base_type = if let Str(s) = base_type {
        syn::parse_str::<Type>(&s.value()).expect(
            "argument to option base in attribute vectorize must \
             be a string convertible to a type",
        )
    } else {
        panic!(
            "argument to option base in attribute vectorized must \
             be a string convertable to a type"
        )
    };

    // Determine the vectorized type
    let vec_type = quote!(Vec <#base_type>).into();
    let vec_type = parse_macro_input!(vec_type as Type);

    // Extract all fields labeled with the vectorize attribute
    let vectorized: Vec<_> = if let Data::Struct(ref s) = ast.data {
        // Process the struct fields
        match s.fields {
            // Named
            Fields::Named(ref fields) => {
                // Accumulate the fields that contain the vectorize attribute
                fields
                    .named
                    .iter()
                    .filter(|field| {
                        field.attrs.iter().any(|a| a.path.is_ident("vectorize"))
                    })
                    .cloned()
                    .map(|field| {
                        let name = field.ident.unwrap();
                        let ty = field.ty;
                        if ty != base_type && ty != vec_type {
                            panic!(
                                "vectorize attribute applied to field {} of \
                                 type {}, but the vectorize attribute \
                                 requires the types to be either {} or {}",
                                quote!(#name),
                                quote!(#ty),
                                quote!(#base_type),
                                quote!(#vec_type)
                            );
                        }
                        (name, ty)
                    })
                    .collect()
            }

            // All other
            _ => panic!(
                "Can only use Vectorize derive with structs that \
                 contain named fields"
            ),
        }

    // Panic when we don't have a struct
    } else {
        panic!("Can only Vectorize derive macro on structs")
    };

    // Accumulate the commands to push the fields into a vector
    let myinto: Vec<_> = vectorized
        .iter()
        .cloned()
        .map(|v| {
            let name = v.0;
            if v.1 == base_type {
                quote!(vec.push(self.#name);)
            } else {
                quote!(vec.extend(self.#name);)
            }
        })
        .collect();

    // Accumulate the commands to take elements from a vector
    let myfrom: Vec<_> = vectorized
        .iter()
        .cloned()
        .map(|v| {
            let name = v.0;
            if v.1 == base_type {
                quote!(self.#name = v.pop().unwrap();)
            } else {
                quote!(self.#name = v.drain(0..self.#name.len()).collect();)
            }
        })
        .collect();

    // Generate the code
    let generics_params = ast.generics.params;
    let generics_where = ast.generics.where_clause;
    let gen = quote! {
        trait Vectorize <#generics_params> #generics_where {
            fn to_vec(self) -> #vec_type;
            fn from_vec(self,v : #vec_type) -> Self;
        }
        impl <#generics_params> Vectorize <#generics_params>
        for #name <#generics_params>
            #generics_where
        {
            fn to_vec(self) -> #vec_type {
                let mut vec = Vec::new();
                #(#myinto)*
                vec
            }
            fn from_vec(mut self,mut v : #vec_type) -> Self {
                #(#myfrom)*
                self
            }
        }
    };
    gen.into()
}

This allows code such as:

// External dependencies
use mylib::{Vectorize};

#[derive(Vectorize, Debug, Clone)]
#[vectorize(base = "T")]
struct MyStruct <T>
where
    T : std::fmt::Debug
{
    #[vectorize]
    x: Vec<T>,
    y: bool,
    z: T,
    #[vectorize]
    w: T,
    s: String,
    v: Vec<f64>,
}

fn main() {
    let foo = MyStruct {
        x: vec![1.2, 2.3, 3.4, 4.5],
        y: true,
        z: 5.6,
        w: 6.7,
        s: "I like pie!".to_string(),
        v: vec![7.8, 9.10],
    };
    println!("{:?}", foo);
    let vec = foo.clone().to_vec();
    println!("{:?}", vec);
    let foo = foo.from_vec(vec.iter().cloned().map(|x| x * 2.0).collect());
    println!("{:?}", foo);
}

which gives:

MyStruct { x: [1.2, 2.3, 3.4, 4.5], y: true, z: 5.6, w: 6.7, s: "I like pie!", v: [7.8, 9.1] }
[1.2, 2.3, 3.4, 4.5, 6.7]
MyStruct { x: [2.4, 4.6, 6.8, 9.0], y: true, z: 5.6, w: 13.4, s: "I like pie!", v: [7.8, 9.1] }
1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.