Iterator returning both refs to Vec elems and ref to instance owned by the iterator

I'm back to Rust after a long while and I'm struggling with lifetimes a bit. A simplified example:

I have an Iterator which alternates between returning references to Vec<T> elements and a default-constructed T which is owned by the Iterator itself.

pub struct AlternatingIter<'a, T> 
where 
    T: Default,
{
    vec: &'a Vec<T>,
    default_value: T,
    // ...
}

The sticking point is implementing Iterator::next() where

fn next(&mut self) -> Option<Self::Item> 

&mut self lifetime clashes with lifetime of Self::Item which is &'a T.

I've tried introducing 'b but I don't know how/if it's possible to express that 'a outlives the &'b mut of the iterator instance when next() is called.

Context: I'm trying to avoid making copies of returned items since in my project, they're a bit heavy to construct and the plan behind the iterator is to use it in a busy loop and if possible, I'd like to avoid using Rc elements. Also, in my project I'm not using Default trait but I construct the "default" item manually and modify it in-place in (some of the) calls of next(), but that's another topic. :smiley:

Playground: Rust Playground

Iterators can't return references to themselves.

You can instead store a reference:

default_value: &'a T,

Or you can skip implementing Iterator and just have a for_each-like method.

1 Like

and ref to instance owned by the iterator

This is impossible in Iterator, because next is &'a self -> Option<Self::Item>, and there's no way to get that 'a into the item type.

You can look up "streaming iterators" or "lending iterators" for more about this problem.

If you've ever wondered why https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.chunk_by has the note about only being IntoIterator and not Iterator, it's because of this limitation. It needs to return an object, then the iterator for that object can return references into that object -- that way it's not trying to return references into the iterator itself.

2 Likes

The Iterator interface guarantees that it's always valid to use .collect():

let collection = iter.collect()

which allows all elements returned by the iterator to exist at the same time, and the collection to continue to exist after the iterator has been destroyed.

If you were allowed to return references to default_value: T, you'd end up with collections of dangling pointers.

2 Likes

Thanks everyone! (Especially @scottmcm, now I know how it's called. :smiley: )

It makes more sense now. I had a feeling I was getting into the self-referential territory.


In the end I'm happy not to use actual Iterator for now. I renamed the class to Cursor and implemented next() as

// before: fn next(&mut self) -> Option<&'a T> 
pub fn next(&mut self) -> Option<&T> 

which does the job for now. I don't really need any of the additional iterator functionality as I use returned references one at a time.

1 Like