How does Iterator.by_ref() allow retaining ownership of the original iterator?

[deedurick]

According to my current understanding of ownership and borrowing, the following code should not compile, but it does. Could someone please point out the flaw in my thinking?

let mut words_iter = ["hello", "world", "of", "Rust"].into_iter();
let words_iter_ref = words_iter.by_ref();

// Take the first two words.
let hello_world: Vec<_> = words_iter_ref.take(2).collect();
assert_eq!(hello_world, vec!["hello", "world"]);
  
// Collect the rest of the words.
// We can only do this because we used `by_ref` earlier.
let of_rust: Vec<_> = words_iter.collect();
assert_eq!(of_rust, vec!["of", "Rust"]);

Why I thought this should not compile (I've number the steps so it might be easier to point out the mistake):

1. words_iter is a std::slice::Iter
2. words_iter_ref is an &mut Iter
3. Calling take on words_iter_ref triggers automatic dereferencing by the compiler, because the take method has self as its first parameter, and not &self.
4. Therefore, take will be called with *words_iter_ref, which is of type *&mut Iter, which is equal to Iter
5. Therefore, take will take ownership of the original iterator owned by words_iter. I.e. a move happens.
6. When calling words_iter.collect() later, the collect method will try to take ownership of a value that was already moved in the call to take before. Therefore the code won't compile.

In other words: it seems to me that by simply doing a reference immediately followed by a dereference, you can completely circumvent the ownership rules!

There must be something I'm not getting. I would appreciate it if someone could help me understand why this code works and thereby improve my conceptual understanding of Rust. Thanks!

[alice]

It's because mutable references to iterators are also iterators, so take is being called with Self = &mut Iter rather than Self = Iter.

[deedurick]

Thanks, this helps. I still don't fully understand this part though

It's because mutable references to iterators are also iterators

Is it a general rule that if T implements MyTrait then &mut T also implements MyTrait, or is this something special about the Iterator trait?

[alice]

No, it's because the standard library contains an impl block for mutable references.

impl<I: Iterator + ?Sized> Iterator for &mut I {
    type Item = I::Item;
    #[inline]
    fn next(&mut self) -> Option<I::Item> {
        (**self).next()
    }
    fn size_hint(&self) -> (usize, Option<usize>) {
        (**self).size_hint()
    }
    fn advance_by(&mut self, n: usize) -> Result<(), usize> {
        (**self).advance_by(n)
    }
    fn nth(&mut self, n: usize) -> Option<Self::Item> {
        (**self).nth(n)
    }
}

[deedurick]

Ahhh, now it all makes sense! This is why the automatic dereferencing doesn't need to happen, because take is implemented straight on the &mut Iterator

[semicoleon]

Dereferencing can never "move" ownership from the perspective of the borrow checker. If you have an &mut T there's no way to get a value of T and invalidate the variable the &mut T was pointing to.

There are a bunch of was to get a T out of the &mut T, but none of them would prevent someone from using the original variable.

[deedurick]

Indeed, it seems that calling a method with receiver type Self on a &mut T leads to error E0507. So even though my thinking went wrong in step 3 already, what I wrote in step 5 would have been wrong as well.

[quinedot]

by_ref is roughly an ergonomic ^[1] alternative to

    let hello_world: Vec<_> = (&mut words_iter).take(2).collect();

And another place you might see this pattern is when you don't want for to totally consume your iterator.

let mut iter = ...;
for item in &mut iter { // or iter.by_ref()
    if condition { break; }
}
// `iter` still usable

---

1. especially when chaining ↩︎