Iterator yielding references to buffer

#1

The standard Lines iterator allocates one string per line. Conversely, with .read_line(), it’s possible to allocate just one buffer and reuse it (playground).

So naturally, I wondered whether I could wrap this in an Iterator interface. After reading up a little bit on streaming iterators, I realized it wouldn’t be so simple. Still, this answer on SO says that even though I can’t return a reference into the iterator struct itself from .next(), I should be able to return a reference into some other structure (see paragraph beginning with “Update:”).

So I came up with this solution, where the Iterator stores a mutable reference to a String, reads data into it, and each call to .next() returns that reference. But this complains about conflicting lifetimes (see messages if you try to compile the example).

Probably the most confusing part is the message (simplified) expected Option<&'a String>, found Option<&String> – how is that possible? The type of the line field of MyLines is defined as &'a mut String, so how can Some(self.line) (where self: MyLines) not be Option<&'a String>?

I would be very grateful for any help in better understanding the problem and whether it’s fixable or not :slight_smile:

Lifetime question regarding iterators
#2

At a high level, this approach won’t work because each time you clear the String it invalidates any previous references given out by the iterator. An Iterator doesn’t remain borrowed after next returns, so you can’t require the previous borrow to end before next gets called again. This is why Iterator can have methods like collect that call next repeatedly while keeping the previous results alive.

The specific error is that &mut T references must be unique and therefore cannot be copied (unlike &T references). Therefore your next method can only borrow the &mut String reference contained in self; it can’t give away copies that outlast self.

What you could do is read the entire file into a String all at once and then iterate over the lines of that String. The existing str::Lines iterator can do this. It works because it has a shared (&str) reference rather than a unique (&mut) reference to the string.

#3

Thank you very much for the reply, I’ve been wrapping my head around it :slight_smile: I think the key piece of information for me is that unique references aren’t Copy, which sounds obvious in hindsight (duh, they wouldn’t really be unique if they were Copy), but I didn’t quite make the connection that returning anything from a function means either copying it or moving it, and I can’t do either here.

(Move of self.line is prohibited because I just have &mut self, not self, and it’s not what I want anyway because I need to keep the iterator struct around; and copy is prohibited because I have &mut String, not &String; sound about right?)

#4

I’m wondering whether the compiler could have been more helpful in terms of the messages displayed… In general, I feel like type inference, lifetime elisions, automatic coercions and dereferencing etc. are awesome and make code much cleaner and more readable when everything works fine, but whenever things go south, I find myself wishing all of these were explicit so that I could reason about them without having to make guesses which might be wrong.

If only the compiler could output an HTML version of my source code with all of these explicitly annotated, lifetimes represented visually etc. :slight_smile: (That’s not to say it doesn’t already do a great job generating intelligible, actionable messages, plus it’s backed by an incredibly helpful community, which of course beats any fancy visualization.)

When writing Rust, I often get stuck wondering whether something I’d like to do is just not possible at all, or whether I’m only doing it wrong. But I guess everything is possible in principle using unsafe, so it’s inherently a hard question and I just need to develop some intuition through experience, so thank you very much again for the feedback and explanations :slight_smile: The part about having to keep previous results alive during an iteration also helped focus my understanding of iterators a lot!

#5

Don’t quote me on this, but I think you can get some of this with the Rust plugin to IntelliJ, which among other things can display the inferred type of your data. In general, IDEs and other “heavyweight” text editors are quite good at clarifying code written in highly implicit languages through interactive visualisations.

#6

Thanks for the tip, I’ll investigate! I’d rather not leave my editor of choice too often, but it makes sense until I’m a little bit more comfortable with Rust :slight_smile: