A better idea for my streaming iterator

help
1

Hi all,

I wrote a dedicated streaming iterator, which is reading a file line by line and returning a mutable structure. My iterator has a next() method, and the way to use it is:

 while let Some(rec) = reader.next() {
   // fancy code here
}

But as soon as I need to use a field of the reader struct, the borrowing rules kick in and my code doesn't compile anymore.

I found a solution by returning a tuple:

 while let Some((stats,rec)) = reader.next() {
   // fancy code here
}

where stats in a clone of the reader fields I'm interested in, that clone coming at the end of my next() method. I didn't find any clever solution.

Is it a standard pattern to return such a tuple in those cases ?

Thanks a lot for your very positive help, as always.

1 Like
2

If the return type of the reader.next() is mutable and owned by the reader (e.g. you are not returning by value) than you can not mutate reader without using RefCell or something like that. Essentially you want interior mutability which exactly what the RefCell is for.

3

@Botev

Indeed, the return of next() is a reference to a struct, owned by a field of Reader. But I don't want to mutate, just access some fields, which I can't.

4

But then why does the reader.next() return mutable struct if you are not going to mutate it?. In general, you can make whatever part of the Reader you mutate into a RefCell. Or if you can write a very simple example of exactly what you are doing would be even better :slight_smile:

5

If that signature is fn next(&mut self) -> TypeWithReference, then it will keep an exclusive borrow even if the returned value is not mutable. There have been discussions about how to "downgrade" references in this sort of situation, but there's nothing you can do for that today.

1 Like
6

In the example I gave, there's no mutation. But sometimes, I might happen when using my (future) crate.

7

It's common create two iterators iter and iter_mut to handle the mutating case separately. The reason for an iter_mut is precisely (I expect) because you can't othewise mutate the data structure while the iterator has a reference to it.

2 Likes
8

On the more technical side, you may be interested in the rust streaming repository.

They're doing experiments with zero-copy streaming. Note that they have a heavy focus on the zero copy part, so expect lifetime wizardry and millisecond performance discussion :slight_smile: