Returning aliased objects alongside iterator aliasing it

help
1

Hey folks! I'm using tree-sitter to parse some Rust source code. I'm trying to write a function that produces an iterator over all the children module items of a node. I have a function that more or less looks something like

pub fn get_node_children_modules<'tree>(
    tree: &tree_sitter::Node<'tree>,
    source_bytes: &[u8],
) -> impl StreamingIterator<Item = ModuleItem<'tree>> {

    const MOD_QUERY_STRING: &str =
        "(mod_item name: (identifier) @ident body: (declaration_list)) \
         @item_mod_body \n(mod_item name: (identifier) @ident) @item_mod";

    let ast_query =
        Query::new(&tree_sitter_rust::LANGUAGE.into(), MOD_QUERY_STRING)
            .unwrap();

    let mut cursor = QueryCursor::new();

    // ... do things with `cursor` ...

    let mod_items = cursor.matches(&ast_query, *tree, source_bytes);

    let capture_unpack = |qm: &QueryMatch| {
        // return objects bound to the lifetime of
        // `cursor` and `ast_query`
    };

    mod_items.map(capture_unpack)
}

Unsurprisingly, this does not work as ast_query and cursor do not live long enough. Instead, I've tried to return a type that also contains cursor and ast_query:

struct ModuleItemIterator<'tree> {
    ast_query: tree_sitter::Query,
    cursor: tree_sitter::QueryCursor,
    iterator: Box<dyn StreamingIterator<Item = ModuleItem<'tree>> + 'tree>,
}

impl<'tree> StreamingIterator for ModuleItemIterator<'tree> { /* */ }

The issue is more or less the same:

error[E0515]: cannot return value referencing local variable `cursor`
   --> src\project_structure.rs:229:5
    |
179 |       let mod_items = cursor.matches(&ast_query, *tree, source_bytes);
    |                       ------ `cursor` is borrowed here
...
229 | /     ModuleItemIterator {
230 | |         ast_query,
231 | |         cursor,
232 | |         iterator: Box::new(mod_items.map(capture_unpack)),
233 | |     }
    | |_____^ returns a value referencing data owned by the current function

error[E0515]: cannot return value referencing local variable `ast_query`
   --> src\project_structure.rs:229:5
    |
179 |       let mod_items = cursor.matches(&ast_query, *tree, source_bytes);
    |                                      ---------- `ast_query` is borrowed here
...
229 | /     ModuleItemIterator {
230 | |         ast_query,
231 | |         cursor,
232 | |         iterator: Box::new(mod_items.map(capture_unpack)),
233 | |     }
    | |_____^ returns a value referencing data owned by the current function

error[E0505]: cannot move out of `ast_query` because it is borrowed
   --> src\project_structure.rs:230:9
    |
143 |   pub fn get_node_children_modules<'tree>(
    |                                    ----- lifetime `'tree` defined here
...
163 |       let ast_query =
    |           --------- binding `ast_query` declared here
...
179 |       let mod_items = cursor.matches(&ast_query, *tree, source_bytes);
    |                                      ---------- borrow of `ast_query` occurs here
...
229 | /     ModuleItemIterator {
230 | |         ast_query,
    | |         ^^^^^^^^^ move out of `ast_query` occurs here
231 | |         cursor,
232 | |         iterator: Box::new(mod_items.map(capture_unpack)),
233 | |     }
    | |_____- returning this value requires that `ast_query` is borrowed for `'tree`

error[E0505]: cannot move out of `cursor` because it is borrowed
    --> src\project_structure.rs:231:9
     |
 143 |   pub fn get_node_children_modules<'tree>(
     |                                    ----- lifetime `'tree` defined here
...
 174 |       let mut cursor = QueryCursor::new();
     |           ---------- binding `cursor` declared here
...
 179 |       let mod_items = cursor.matches(&ast_query, *tree, source_bytes);
     |                       ------ borrow of `cursor` occurs here
...
 229 | /     ModuleItemIterator {
 230 | |         ast_query,
 231 | |         cursor,
     | |         ^^^^^^ move out of `cursor` occurs here
 232 | |         iterator: Box::new(mod_items.map(capture_unpack)),
 233 | |     }
     | |_____- returning this value requires that `cursor` is borrowed for `'tree`

It wouldn't be an issue to instead eagerly evaluate the iterator and just return a vector. I'd just like to ask whether something like this is possible for educational purposes. Is this even a good idea? Would it be possible to implement it by pinning the cursor and ast_query objects, and defining the iterator field to be MaybeUninit<Box<_>>?

2

Since this approach is inherently self-referential, I've decided to instead try refactoring this function into an intermediate type that contains cursor and ast_query (alongside tree and source_bytes) and implement a "pretend" IntoIterator method iter_mut for this type (StreamingIterator does not implement Iterator).

Still, the issue persists, as the closure I use in the return object mod_items.map(capture_unpack) isn't convinced by the lifetime annotations I've added to the signature of the function:

 pub fn iter_mut<'qc>(
        &'qc mut self,
    ) -> impl StreamingIterator<Item = ModuleItem<'qc>>
        where 'tree: 'qc, 'src: 'qc

error: lifetime may not live long enough
   --> src\project_structure.rs:230:17
    |
189 |         let capture_unpack = move |qmatch: &QueryMatch| {
    |                                    ------             - return type of closure is ModuleItem<'2>
    |                                    |
    |                                    has type `&QueryMatch<'_, '1>`
...
230 |                 ModuleItem::Definition(capture_node)
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
3

I don’t have time to look deeply into this atm, but given that you’re aware of the phrase “self-referential”, have you tried a crate like yoke?

Perhaps the problem also involves lending iterators?

4

Thanks for the reply!

It indeed seems the problem was related to lending iterators, at least partially. It turned out that I gave the compiler insufficient lifetime information, and it couldn't determine that the object passed to the closure lived long enough. Explicitly annotating QueryMatch with 'qc as below helped fix the problem:

fn iter_mut<'qc>(
        &'qc mut self,
    ) -> ... {
    
    let capture_unpack = move |qmatch: &QueryMatch<'qc, 'tree>|
        { /* ... */ };

    // ...
}

For future reference: I did effectively heed the advice in that post about self-referential types and take out the types referred to into their own types.