IntoIterator and reference on array

1

Hello,

I have two codes:

for item in ITEMS {
     ...
}

for &item in ITEMS {
     ...
}

ITEMS is:

const ITEMS: &[&str] = &["ITEM_1", "ITEM_2"];

In the first code item is &&str and in the second code &item is &str.

With a reference (second code with &item) it can be seen as a tuple. The array can be seen as "unpack" I suppose (as to unpack a tuple) ? And &item is the reference of each element in the array (so &str) ?

Can the explanation be found here ?

Thank you very much in advance for any help

2

note, the syntax &item is not declaring a reference, but actually does the opposite: it dereferences the value yielded by the iterator, which is &&str,

IntoIterator is implemented for slices like this:

impl<'a, T> IntoIterator for &'a [T] {
    type Item = &'a T;
    //...
}

in your example, ITEMS is a slice of &[&str], so T is &str, which means the iterator will yield values of &T, which is &&str.

then the loop variable binding item is created just like a normal variable declaration. when you use item, it binds to &&str. when you use &item, then item binds to &str.

formally, it is called a reference pattern.

in rust, a variable binding is created by pattern matching, the most common form is the irrefutable let statement as we know:

let some_variable = some_expression;

but this is just a one special case of pattern matching, where the pattern is just a single identifier (the identifier pattern). the full list of valid patterns in a let statement can be found here

in your example, the desugared code of the for-loop creates the binding like this (illustratively):

{
    let mut __iter = ITEMS.into_iter();
    loop {
        match __iter.next() {
            Some(__next_value) => {
                // note: the type of __next_value is `Iterator::Item`
                // which is `&&str` in this case, so the pattern
                // `&item` is matched against `&&str`, which makes
                //  `item` to bind to a value of `&str`
                let &item = __next_value;
                ...
            }
            None => break,
        }
    }
}
3

I'm really sorry, but could you please explain ? I'm not sure to understand :confused:

Do you mean here ?

I'm really sorry but could you please explain ? :confused:

Which implementation shows whether it is a &str or a &&str ?

That I understood

Thank you for your great help :slight_smile:

4

I am not sure if this will help, but it is similar to manipulating equations on math. You can write

  • x = a + 1 or
  • x + 1 = a

but the two are wery different. To see this we can subtract 1 from both sides of the second equation to obtain x = a - 1. So while the first one "adds 1 to a", the second one "subtracts 1 from a" despite having a + sign in it (it is on the other side).

Similarly in Rust you can write:

let x = &a;

or

let &x = a;

If you "dereference both sides" of the second one, you get:

let x = *a;

So you can see that when the & sign is on the left side of the "equation" it dereferences the right side.

this one:

impl<'a, T> IntoIterator for &'a [T]
    type Item = &'a T
    type IntoIter = Iter<'a, T>

The type of your ITEMS is &[&str], so by unifying &[&str] = &[T] we obtain T = &str and by substituting that into the Item we obtain Item = &T = &&str.

5

I recommend learning about pattern matching in three stages.

Stage 1: Patterns are the duals of expressions. The motivation for that article is exactly the question in your OP! The article explains how when you match

let &item = a_value; // where a_value has type &&str
//  -^^^^                                      -^^^^
//   item has type `&str` (`&&str` with one `&` destructured)

The & destructures ("unpacks") the outer reference.

I suggest you read the article and come back with any remaining questions.

That's enough for this topic, but the other stages are

Stage 2: Learn the ref and ref mut modifiers. Those let you create a reference to some place that's within the scrutinee.

fn example(o: Option<String>) {
    if let Some(ref inner) = o {
        // inner is a `&String`
    }
}

Stage 3: Learn about binding modes, which can be considered to desugar to patterns that never change the default binding mode.[1]

fn example(o: Option<String>) {
    // Desugars to what was in the last code snippet
    if let Some(inner) = &o {
        // inner is a `&String`
    }
}

  1. You can use this lint to detect when binding modes are being used. ↩︎

6

as I said, you need to understand pattern matching, then this becomes obvious, but the meaning of this exact sentence can be explained by @Tom47's algebra metaphore:

// this code
let &variable = expression;
// and this
let variable = *expression;
// creates the same variable binding

note, these two are not always identical, due to the complication how the * operator works[1], but it's suffient as a demo and over-simplified understanding for the concept of reference patterns.

yes, that's implementation code I was refering to.

let's use the algebra metaphore again:

// this `for` loop:
for &item in ITEMS {
    ...
}
// is equivalent to this:
for item in ITEMS {
    let item = *item;
    //...
}

I hope the example helps. but ultimately, as every suggests, the key is to understand the whole picture of pattern matching, the &item in this example is just a special case of it: the reference pattern.

you already found the impl code, it's this one in the docs

the standard library does NOT implements IntoIterator for slice of individual types, e.g. slices of bool, slices of i32, slices of &str, or slices of Box, instead, it implements for all slice types generically: impl<'a, T> IntoIterator for &'a [T]. let's not talk about the lifetime 'a for now, here T is a type parameter.

whatever slice type you end up use, you substitute T with the concrete type. in this example, the actual type is &[&str], so T must be &str, in order to make &[T] and &[&str] the same type. then you read the impl block: Item = &'a T, since T is &str, &T must be &&str (again, ignore the lifetime for now).

  1. e.g. if expression is not itself a reference, but a type implementing Deref, the first pattern will fail to compile, but the second line is ok ↩︎

7

I get your confusion :< the syntax for &var in is indeed confusing. It speaks like taking the memory address of each element and we need to dereference it afterwards, because the syntax & is the language of "taking the memory address" in Rust. But it suddently becomes "take the memory address and dereference it". So to avoid confusion like this, I think syntax should represent the last action. So it should be

for *name in data {
    // it is clear, name is already value not memory address, because *name returns value
    // so user clearly can know they can use it directly without dereference again, because the keyword `*` already communicate it is already value that is taken from a memory address, not it is still memory address
    // user can clearly know they do not need to dereference (*) again
    println!("{}", name);
}

Others have mentioned the technical detail, I want to provide a different perspective from high level explanation focusing on what is going on, not the implementation design

Let's start with your const variable. It is

const name = &[ &str ]

Which is a slice, aka a fat pointer, fat pointer is pointer that has runtime len metadata. The slice's pointer points to multiple T in contigous memory but only within certain range, the range is 0 until the value of len. Here is the illustration

struct name<T> {
    ptr: *const T,
    len: usize
}

It is different than &[Type; Length]. It has compile time length. Aka it doesn't has runtime len metadata. A pointer that doesn't has any runtime metadata is called thin pointer. The pointer points to fixed size contiguous memory. Here is the illustration

struct a<T> {
    ptr: *const T
}

Since you created a const value, intended to never be mutated again, you should use thin pointer variant instead. Because it does not have runtime len metadata. Suprisingly the compiler doesn't optimize slices that are guaranteed never change again, like your code above

const ITEMS: &[&str] = &["ITEM_1", "ITEM_2"];

Despite it is declared as const, with literal values, I checked the assembly, it never be optimized to thin pointer. I will search does the two has different optimization capabilities first :v I will share the result of it :>

Now, let's move to the iterator thing

Your variable is

const ITEMS: &[&str] = &["ITEM_1", "ITEM_2"];

The values of your variable above are multiple &str. Remember &str is another pointer that points to a static data embedded in the binary. The &str is independent from the iterator, iterator only communicates directly with the container, aka the &[...], so you can ignore &str to avoid the confusion of double &&. Let's call it type instead. So your variable is &[type]

Let's talk about the normal for in first

for val in data

Let's call val is the left type, data is the right type

The default behavior of for in iterator is

  • if right type is value type + implement copy -> val becomes owned type that copy the element

  • if the right type is value type + does not implement copy -> val becomes owned type that move the element

  • if the right type is reference type like &[...], &Vec, etc -> val becomes reference that points to the element (borrow)

Additional :
.iter() make value types become reference types that can be used for reference based iterator. Like

let data = vec![1, 2, 3];
// .iter() makes owned value become reference that can be used for iteration without copying or moving the element
for val in data.iter() {

}

// you can still use data here

Now for &val in data just extends the reference based iterator above becomes iterator by reference + auto dereference. It can only be used in reference based iterators, because ofc no need to dereference in an already value type

So when you encounter

for &val in data.iter() {
    let get = val;
}

Remember it is just a sugar for

for val in data.iter() {
    let get = *val;
}

It removes the need of calling * (dereference) each time need to access the value

Or you can keep this synonym for easy understanding. for &val in data is just for *val in data. It auto dereference

8

So if I unserstand correctly, this code:

let &x = a;

unreferences a ?

Great ! Thank you for pointing me the implementation :slight_smile:

Thank you for your precious help and answer

9

This is just excellent ! Now it makes sense ! Thank you very much !

I will of course read the article

What do mean about the stages ? Do you mean that I should read stage 2 then stage 3 ? Is it part of my question ? Or is it some more knowledge that I should know ?

Thank you for your precious help and answer

10

Thank you very much ! Now it's like I don't feel so stupid ! Thank you for understanding my confusing.

If it's copyable I suppose ?

Now I think I'm getting to start to understand where you're getting at.

I'm really sorry but I'm not sure to understand the:

 let get = *val;

Do you mean that it's a way to access the "raw" / "real" type (&str in my case) of each value in my slice ?

11

I mean get a good understanding of stage 1 first (which the article explains well), then afterwards of stage 2, and then finally of stage 3 (which can be understood in terms of stage 3).

Taking it all in at once can be confusing -- especially stage 3 IMO, which is invisible[1] -- but they are more approachable one stage at a time, and build on the previous stages.

  1. that's why I mentioned the lint ↩︎

12

Yea, &str implement copy. So the key is, as long as the right type is by value and implement copy, the left type will copy the element. To know if it implements copy, afaik there is no easy fast way except reading the documentation, browsing in google or simply ask AI, or use manual trick

let a = 2;
let b = a;

// if a implement copy, this will not cause error use of moved value
println!("{}", a);

*val is independent from the iterator thing. It is just an example of usage. *val is syntax for accessing the value of pointer. You can think of it like it is unwrapping the pointer so that it is not just memory address anymore, but the value that is pointed by said address

let a = 10;

// a is already value type, it is already normal variable so you can use it directly 

println!("{}", a);

let pointer = &raw const a;

// pointer and reference is a variable containing a memory address, not a value. * is the syntax to unwrap the value of memory address in Rust, and many other languages like C, C++, etc if you have used it. If you do not dereference a memory address variable, it will only return a memory address because it is memory address in the first place

println!("{}", *pointer);

// you need to dereference in each individual access if you don't store the result of dereference in variable like `let result = *memory_address`
println!("{}", *pointer);

The correlation is, since for &val in data already auto dereference under the hood as I typed in the previous comment, the variable val here already holds real value, no longer memory address, so you just access it like a normal variable value afterwards, no longer need to dereference in each individual access

13

Okay, I understand, thank you

14

Thank you for your detailed response.

I'm sorry, but I what I meant was, "why is it writtent this way ?" Why is it dereferencing the value ? Like I asked, is it a way to copy the "raw" / "real" value ?

I completely understand the dereferencing.

15

Do you mean the original topic, for &var in. Or *var inside the loop that I typed before?

16

When you meant:

So from the second loop

17

So I won't find any implementation I suppose ?

That really help thank you !