Proc macros: How to update stack variables when inside a Iterator map

I've run into this issue too many times and I've finally decided to ask help for it.
Usually when using quote! inside iterables, we use map to map fields of a struct / variants of an enum into something that produces a TokenStream. So generally one writes something like this:

let counters = input.variants.iter().map(|variant| {
		let variant_ident = ident_to_snake_case(variant.ident.clone());
		let fn_count = quote::format_ident!("{}_count", variant_ident);
		orginal_and_sc.push((variant.ident.clone(), fn_count.clone()));
		quote! {
			fn #fn_count (&self) -> usize;
		}
	});

Now the problem is original_and_sc is a stack variable, and it's borrowed until the end of the function where I will be using counters in the final token stream like #(#counters)*. So now I can't do anything useful with my original_and_sc variable and this is very frustrating. Is there a better pattern to follow when dealing with iterators and maps in proc-macros?

Collect the iterator into a Vec<TokenStream> before you assign it to counters

let counters: Vec<_> = input.variants.iter().map(|variant| {
		let variant_ident = ident_to_snake_case(variant.ident.clone());
		let fn_count = quote::format_ident!("{}_count", variant_ident);
		orginal_and_sc.push((variant.ident.clone(), fn_count.clone()));
		quote! {
			fn #fn_count (&self) -> usize;
		}
	}).collect();

Alternatively wrap orginal_and_sc in a RefCell so you don't capture a &mut to it in the closure.

1 Like

What you are doing is an anti-pattern. You are not supposed to mutate stuff in the projection of map. Instead of doing everything twice or collecting everything into a large token stream, you could collect the identifiers only in a first pass:

let orginal_and_sc: Vec<_> = input.variants
    .iter()
    .map(|variant| {
        let variant_ident = ident_to_snake_case(variant.ident.clone());
        let fn_count = quote::format_ident!("{}_count", variant_ident);
        (variant.ident.clone(), fn_count)
    })
    .collect();

let counters = original_and_sc.iter().map(|(_, fn_count)| {
    quote! {
        fn #fn_count (&self) -> usize;
    }
});

Boom, mutation gone, mutable-borrowing-forever gone, lazy mapping achieved.

1 Like

I would suggest since counters here is also borrowing original_and_sc through iter() it's best if we add a collect() here as well in the spirit of the solution.

I don't understand what you mean. Wasn't the whole point of the question that you wanted to keep the lazy map?

My question was often times inside maps I'd find myself extracting information to stack variables and the mutable borrows in maps that would be later reused in tokenstreams would prevent me from doing that. Since then I've learned that a Vec can be equally reused in place of a Map and so if we were to just collect() on the maps that reference any stack variable, we could avoid that.

I know this reads very confusingly, and your answer does help address my issue, but I use original_and_sc in more places than I've shown here and I think if I don't add the collect() in

let counters = original_and_sc.iter().map(|(_, fn_count)| {
    quote! {
        fn #fn_count (&self) -> usize;
    }
});

I'd get another borrow checker error if I reuse original_and_sc.

That depends. If you only borrow original_and_sc immutably in the subsequent code (as long as the map exists), then it's fine, because you are allowed to have several immutable borrows at the same time.

Ah I see. Thanks for pointing that out.