How to return a reference to struct initialized data in an iterator

Hey all!

I really wonder how this is done in Rust. I have a struct that holds a vector, like so:

struct Wow {
    v: Vec<usize>,
    i: usize,
}

In want to be able to populate this vector, v, during each iteration (next call) in an Iterator and return a reference to v so I prevent cloning it. The following code describes what I want:

struct Wow {
    v: Vec<usize>,
    i: usize,
}

impl Wow {
    fn new() -> Self {
        Self {
            v: Vec::with_capacity(100),
            i: 0,
        }
    }
}

impl Iterator for Wow {
    type Item = Vec<usize>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.i > 5 {
            None
        } else {
            self.v.push(10);
            self.i += 1;
            Some(self.v.clone()) // NOTE but then not cloning here
        }
    }
}

fn main() {
    let mut x = Wow::new();
    for y in &mut x {
        println!("{:?}", y);
    }
}

My actually code is of course more meaningful than this :slight_smile: I really wonder how, or if at all, I can just return a reference to v. We know it won't be mutated before the next iteration starts. I tried boxing the vector but without success.

What is the idiomatic way of doing this in Rust?

Thanks a lot!

The interface of Iterator guarantees that this code always works:

let a = iter.next();
let b = iter.next();
drop(iter);
use_both(a, b);

so what you're trying to do is intentionally made impossible by the definition of Iterator.

You won't be able to implement Iterator this way. You can implement something else, like a streaming iterator.

4 Likes

Or make two different structs, Wow and WowIter, with WowIter borrowing from Wow, like Vec (essentially, [T]) does.

1 Like

As explained above Iterator can’t work for this. One possible and simple workaround for some use-cases may be to just provide a simple for_each-style method and call it a day. In the case of Wow, that also means that all the additional state being held in a struct becomes unnecessary, and all can be a simple for_wow function

fn for_wow(mut f: impl FnMut(&[usize])) {
    let mut v = vec![];
    for _ in 0..6 {
        v.push(10);
        f(&v);
    }
}

fn main() {
    for_wow(|y| {
        println!("{:?}", y);
    });
}

Naturally, this doesn’t work in all cases, especially if you want your “loop body” to support early exiting or async or interleaved iteration with other wow's, or there’s generally need features like iterator combinators, etc…

2 Likes

Another option could be using clone-on-write:

use std::sync::Arc;

struct Wow {
    v: Arc<Vec<usize>>,
    i: usize,
}

impl Wow {
    fn new() -> Self {
        Self {
            v: Arc::new(Vec::with_capacity(100)),
            i: 0,
        }
    }
}

impl Iterator for Wow {
    type Item = Arc<Vec<usize>>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.i > 5 {
            None
        } else {
            Arc::make_mut(&mut self.v).push(10);
            self.i += 1;
            Some(Arc::clone(&self.v))
        }
    }
}

fn main() {
    let mut x = Wow::new();
    for y in &mut x {
        println!("{:?}", y);
    }
}

This way, in use-cases like the main above, y is dropped before the next iterator element is created, so the Arc will be unique again, and Arc::make_mut needs not to clone the Vec. Use-cases like @kornel's

let a = iter.next();
let b = iter.next();
drop(iter);
use_both(a, b);

are still supported, but this will result, at run-time, to the decision that the Vec will need cloning. (One downside here in particular is that the Vec’s capacity won’t be remembered if make_mut ever does a clone, as Vec::clone does not preserve capacity.)

Here’s my attempt to keep the capacity through Arc::make_mut-like functionality. It’s not very pretty… alternatives would include e.g. a wrapping newtype with custom Clone… anyways, click to see the code.
fn make_mut_preserving_capacity<T: Copy>(x: &mut Arc<Vec<T>>) -> &mut Vec<T>
// also, without dissociating weak pointers, but that mostly won't matter
{
    if Arc::get_mut(x).is_some() {
        Arc::get_mut(x).unwrap()
    } else {
        let mut v = Vec::with_capacity(x.capacity());
        v.extend(&x[..]);
        *x = Arc::new(v);
        Arc::get_mut(x).unwrap()
    }
}

impl Iterator for Wow {
    type Item = Arc<Vec<usize>>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.i > 5 {
            None
        } else {
            make_mut_preserving_capacity(&mut self.v).push(10);
            self.i += 1;
            Some(Arc::clone(&self.v))
        }
    }
}
2 Likes

That explains it :sweat_smile: thanks! Being new, is there a way I could see these rules?

These are not separate rules. They are the byproduct of a few, very simple borrow checking rules. You have to desugar the lifetimes of functions to see that the return type is independent of the (elided, implicit) lifetime parameter of self.

2 Likes

Aaah this looks interesting as well!

I basically read through the bytes of a mmap, parse some integers, populate two vectors and "return" those. Since I don't want to re-alloc the vectors for every parsed line I thought of using the iterator. So I think using your for_each approach would work for this. Nice to see an example of fnMut, was wondering what that looked like in practice

Here's another way you could implement it. This is effectively the streaming/lending iterator pattern cited before.

struct Wow {
    v: Vec<usize>,
    i: usize,
}

impl Wow {
    fn new() -> Self {
        Self {
            v: Vec::with_capacity(100),
            i: 0,
        }
    }
    
    // Note: The `&[usize]` has a lifetime that is implicitly connected
    // to the one of &mut self due to lifetime elision rules.
    // That is, it is equivalent to:
    //
    // fn next<'a>(&'a mut self) -> Option<&'a [usize]>
    //
    // With `Iterator` this is not possible to do: the `&[usize]` would have
    // to be defined in the `type Item` which cannot name the lifetime of
    // the `&mut self` in the `next` method and thus depend on it.
    fn next(&mut self) -> Option<&[usize]> {
        if self.i > 5 {
            None
        } else {
            self.v.push(10);
            self.i += 1;
            Some(&self.v)
        }
    }
}

fn main() {
    let mut x = Wow::new();
    // No support for `for` loop though,
    // you'll need to use `while let`
    while let Some(y) = x.next() {
        println!("{:?}", y);
    }
}
2 Likes

Thanks! That actually looks even simpler than implementing the Iterator

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.