Returning references to an internal buffer from an Iterator

I have a situation where I have an Iterator which should iterate over a file and return the file block-by-block together with the block's hash. It bothers me to have to reallocate buffers for each iteration, so I would like to reuse them and just return references to some internal buffers stored in the iterator.

I put together a simple illustration on the playground, and it causes:

error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
  --> src/lib.rs:62:15
   |
62 |         hash: &self.hash,
   |               ^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the anonymous lifetime defined here...
  --> src/lib.rs:45:11
   |
45 |   fn next(&mut self) -> Option<Self::Item> {
   |           ^^^^^^^^^
note: ...so that reference does not outlive borrowed content
  --> src/lib.rs:62:15
   |
62 |         hash: &self.hash,
   |               ^^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'a` as defined here...
  --> src/lib.rs:42:6
   |
42 | impl<'a> Iterator for FileBlockIterator<'a> {
   |      ^^
note: ...so that the types are compatible
  --> src/lib.rs:45:44
   |
45 |     fn next(&mut self) -> Option<Self::Item> {
   |  ____________________________________________^
46 | |     let n = match self.reader.read(&mut self.buffer) {
47 | |       Ok(n) => n,
48 | |       Err(e) => {
...  |
68 | |     }
69 | |   }
   | |___^
   = note: expected `<FileBlockIterator<'a> as Iterator>`
              found `<FileBlockIterator<'_> as Iterator>`

When I tried to do this a while back I recall the compiler said that I needed to use generic associated types. I rewrote this to try and see if I could work out how to use them, but I don't remember what I did differently and I got stuck with this instead.

The problem is that you are confusing the two lifetimes named 'a. The one that is placed on your struct definition denotes the lifetime of the reader reference. However, that has nothing to do with the lifetime of the internal buffer. That's tied to the lifetime of self, but the signature of the Iterator trait doesn't let you express this constraint.

You should instead mimic what standard containers do and create a separate iterator type that contains a borrowed (sub-)slice of the buffer.

1 Like

Remember that collect() exists, so every element of the iterator must remain valid at the same time, and even after the iterator itself is destroyed. So you can't lend any data from the iterator object itself. It can only reference data stored outside of itself.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.