Iterator over references that live only for one iteration

1

Hi all,

I'm trying to write an Iterator that returns references that will only be valid for one iteration, similar to this minimal example:

struct TempIter {
    num : Box<u64>
}

impl Iterator for TempIter {
    type Item = &u64;

    fn next(&mut self) -> Option<Self::Item> {
        self.num = Box::new(*self.num + 1);
        Some(self.num.as_ref())
    }
}

impl TempIter {
    fn new() -> Self {
        Self {
            num : Box::new(0)
        }
    }
}

fn main () {
    for t in TempIter::new().take(42) {
        println!("{}", t);
    }
}

Rust Playground

The compiler complains about the Item not having a lifetime, and I realize what the problem is, but I cannot seem to figure out which lifetime annotations to put in there, and where to put them.

My real-world use-case is to implement an iterator for the rosbag api, similar to the example in the linked readme but with an iterator instead of a nested loop. That library is used to read serialized messages out of large files in the rosbag format, and internally works by memory mapping subsequent chunks of that file, since the file would be too big to load it at once. Loading the next chunk unloads the current one, and that's why the references' lifetime must be limited to the current iteration. I'm aware that I could just use clone and hand out owned structs, but since a regular use case is having many messages to iterate over and only being interested in a few of them, I'd like to see whether using references instead brings a performance improvement.

2

This isn't possible with Iterator. The signature of Iterator::next() says that the lifetime of the returned value is independent of the borrowing of self.

You'll have to write a free function instead.

3 Likes
3

It might make sense to write a custom smart pointer type that is usually a pointer to the current page, but gets replaced with an owned version of the message when the backing page gets replaced. This would involve some extra bookkeeping in the rosbag itself, but would let your iterator yield something that acts like it’s owned, but doesn’t require cloning in the common case.

1 Like
4

Will this be true for the next Rust release with the initial GAT support as well?

5

hm, yes, but that would have weird corner cases. there are always multiple messages in one chunk, and if I kept smart pointers to many messages in many different chunks, the app will run out of memory even though the messages themselves would fit into ram.

6

Yes. As I said, this is not about any missing language feature, this is about the very signature of Iterator.

GATs merely make it possible to create a LendingIterator trait with a different signature. You still won't be able to implement the old Iterator for your type discussed above.

2 Likes
7

It will, since Iterator have to be collectable, that is, everything it yields must be able to

  • coexist with each other (e.g. being put into the same vector),
  • continue to exist after the iterator itself is gone.
3 Likes
8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.