Cannot find value `X` in this scope when return variables inside quote!

[Anachoreta]

In my proc macro I have two very similar derives, so I need to avoid code duplication so I moved some logic into separate struct:

struct FieldParser {
    fields_names: Vec<Option<Ident>>,
    fields_values: Vec<TokenStream2>,
}

impl FieldParser {
    pub fn parse(fields: Fields, data: &Vec<u8>) -> Self {
        let Imports { binary_converter, .. } = Imports::get();

        let mut reader = Cursor::new(data);

        let fields_names = fields.iter().map(|f| {
            f.ident.clone()
        }).collect::<Vec<Option<Ident>>>();

        let fields_values = fields.iter().filter_map(|f| {
            let ident = f.ident.clone().unwrap().to_string();
            let field_type = f.ty.clone();
            let ident = format_ident!("{}", f.ident.as_ref().unwrap());

            if f.attrs.iter().any(|attr| attr.path.is_ident("dynamic_field")) {
                Some(quote! { Self::#ident(); })
            } else {
                Some(quote! { #binary_converter::read_from(&mut reader).unwrap() })
            }
        }).collect::<Vec<_>>();

        Self {
            fields_names,
            fields_values,
        }
    }
}

but when I use this struct inside quote! of my proc macro, I got an error:

error[E0425]: cannot find value `fields_names` in this scope
  --> proc_macro\src/lib.rs:91:24
   |
91 |                     #(#fields_names: fields_values),*
   |                        ^^^^^^^^^^^^ not found in this scope

This is my proc macro code:

#[proc_macro_derive(WorldPacket, attributes(options, dynamic_field))]
pub fn derive_world_packet(input: TokenStream) -> TokenStream {
    let ItemStruct { ident, fields, attrs, .. } = parse_macro_input!(input);
    let Imports { cursor, binary_converter, deflate_decoder } = Imports::get();

    let mut is_compressed = quote!(false);
    if attrs.iter().any(|attr| attr.path.is_ident("options")) {
        let attributes = attrs.iter().next().unwrap();
        let attr: Attributes = attributes.parse_args().unwrap();
        is_compressed = quote!(attr.compressed.value);
    }

    let output = quote! {
        impl #ident {
            pub fn from_binary(buffer: &Vec<u8>) -> Self {
                let FieldParser { fields_names, fields_values } = FieldParser::parse(#fields, buffer);

                Self {
                    #(#fields_names: #fields_values),*
                }
            }
        }
    };

    TokenStream::from(output)
}

Could somebody explain, what I did wrong and how to fix this issue ?

[H2CO3]

You are mixing up macro compilation and macro expansion time.

When a #metavariable is encountered by quote!, it is substituted into the resulting token stream right away, from the surrounding macro code, when your macro is being compiled. That's the point of quoting.

But you have no fields_names or fields_values variables declared. Those declarations are inside the quote, so they'll be emitted to the expanded code, which is apparently not what you want.

You seem to be referring to all sorts of other variables only found in the generated code from within the quote, as if they were meta-variables (e.g. buffer). From this code alone, this looks like you are completely mixed up between compilation and expansion time, and I can't tell what you are actually trying to achieve.

[Anachoreta]

Next code works for me:

#[proc_macro_derive(WorldPacket, attributes(options, dynamic_field))]
pub fn derive_world_packet(input: TokenStream) -> TokenStream {
    let ItemStruct { ident, fields, attrs, .. } = parse_macro_input!(input);
    let Imports { cursor, binary_converter, deflate_decoder } = Imports::get();

    let mut is_compressed = quote!(false);
    if attrs.iter().any(|attr| attr.path.is_ident("options")) {
        let attributes = attrs.iter().next().unwrap();
        let attr: Attributes = attributes.parse_args().unwrap();
        is_compressed = quote!(attr.compressed.value);
    }

    let fields_names = fields.iter().map(|f| {
        f.ident.clone()
    }).collect::<Vec<Option<Ident>>>();

    let output = quote! {
        impl #ident {
            pub fn from_binary(buffer: &Vec<u8>) -> Self {
                let mut reader = #cursor::new(buffer);

                Self {
                    #(#fields_names: #binary_converter::read_from(&mut reader).unwrap()),*
                }
            }
        }
    };

    TokenStream::from(output)
}

but I need to replace:

#binary_converter::read_from(&mut reader).unwrap()),*

with smth like next:

if f.attrs.iter().any(|attr| attr.path.is_ident("dynamic_field")) {
	Some(quote! { Self::#ident(); })
} else {
	Some(quote! { #binary_converter::read_from(&mut reader).unwrap() })
}

in other words, I want to call method which has same name as field if field marked as #[dynamic_field], otherwise I need to call binary_converter.

For example:

#[derive(WorldPacket, attributes(dynamic_field))]
struct Test {
	#[dynamic_field]
	field1: u8,
	field2: u16,
}

impl Test {
	fn field1() {}
}

[H2CO3]

Whatever code you want to generate, you'll need to emit outside the quote. To be clear, you can't have quote generate different code based on runtime differences in the code invoking the macro. Is that something you need?

[Anachoreta]

Well, could you tell if smth like this possible to implement ? I need to have access to specific ident inside quote:

let mut dynamic_fields: Vec<Ident> = vec![];
for field in fields.iter() {
	let ident = field.ident.clone().unwrap().to_string();
	let field_type = field.ty.clone();
	let ident = format_ident!("{}", field.ident.as_ref().unwrap());

	if field.attrs.iter().any(|attr| attr.path.is_ident("dynamic_field")) {
		dynamic_fields.push(ident);
	}
}

let output = quote! {
	impl #ident {
		pub fn from_binary(buffer: &Vec<u8>) -> Self {
			let mut reader = #cursor::new(buffer);

			Self {
				#(#field_name: {
					if #dynamic_fields.contains(#field_name) {
						Self::#field_name()
					} else {
						#binary_converter::read_from(&mut reader).unwrap()
					}
				}),*
			}
		}
	}
};

[H2CO3]

That's exactly what I'm saying: you need to move the logic of the if statement out of the quote. You don't want to generate code that contains an if; you want to generate different snippets based on a condition that you macro knows. I.e., instead of

quote! {
    if #condition … {
        #stuff
    } else {
        #other
    }
}

you want

if cond {
    quote!{ … #stuff … }
} else {
    quote!{ … #other … }
}

[Anachoreta]

I afraid I do not see the way how to do this since I need to have access to buffer param which is passed into generated by quote! method.

Probably you could give me a hint ?

[H2CO3]

The buffer parameter only exists at the compilation time of the code invoking your macro. It's simply impossible to make code generation decisions based on it in your macro.

But you don't seem to need it, anyway. You seem to want to check if the particular field being expanded is marked. That's none of the business of the generated code. This decision must be made in your macro, while you are generating the code. You probably meant to write this:

let initializers = field_names
    .iter()
    .map(|field_name| {
        if dynamic_fields.contains(&field_name) {
            quote!{ Self::#field_name() }
        } else {
            quote!{ #binary_converter::read_from(&mut reader).unwrap() }
        }
    });

let output = quote! {
    impl #ident {
        pub fn from_binary(buffer: &Vec<u8>) -> Self {
            let mut reader = #cursor::new(buffer);
            Self { #(#field_names: #initializers),* }
        }
    }
};

As an aside, &Vec<_> as a parameter is harmful (it's no more powerful than &[_] and it forces the caller to allocate a new vector if they have something else that would coerce to a slice just fine). Make that argument a &[u8] instead.

[Anachoreta]

Thank you very much, this works ! It was a bit unclear for me how TokenStream works, but now I see.

[H2CO3]

This doesn't have anything to do with "how TokenStream works". A token stream is just a somewhat-lexed-and-parsed representation of text (the raw source string is broken up into lexemes and parenthesis balance is checked). You'd have encountered the exact same problem and solution if macros operated on raw Strings or fully-parsed AST nodes or whatever.

Instead, this has everything to do with the difference between the various phases at which your macro code is processed:

1. Your macro's code is compiled. It is compiled to code that emits code. It can't, obviously, access runtime values of code it is just about to emit.
2. Your macro's compiled code is invoked by the compiler. This results in the code that your macro emits.
3. The output of your macro (the generated code) is pasted into the source code that invoked your macro.
4. The source code containing the results of your macro expansion is compiled.
5. The compiled program, with the compiled version of the macro-emitted code is run by whoever you give the executable to.

At steps 4 and 5, your macro is long gone. It has already been compiled and invoked, and it hasn't the faintest choice of altering the emitted code at this point.