Deref is a misleading name

community
1

Having worked with Rust for a couple of years, I found myself occasionally implementing Deref on some custom types, e.g. if they represent newtype wrappers around other types. I have never questioned the terms like Deref for the trait, its method deref() or Deref coercion. However, after having dug into the implementation and actual actions that the trait's method performs, I believe that deref is a horribly misleading and unfit name for the trait and so is deref coercion.

The trait does not dereference

The first and foremost thing that struck me is, that the trait Deref, respectively its method deref() does not perform dereferencing at all. You can see this clearly by its signature.

We define a type to "dereference" to via type Target;. The method fn deref(&self) -> &Self::Target; then converts a reference to the implementor (&self) to a reference of the target type. That's right Deref::deref() returns a reference, not a dereferenced type.

While the documentation clearly states how Deref works and what its intentions are, this goes against my intuition of a trait being named after the action it performs. Deref does not dereference. It changes how dereferencing a type behaves. Other traits in the standard library are named after what they do. E.g. AsRef returns a reference to target type, Add adds stuff. A better name for it may have been RefMap because it actually maps one reference to another type of reference, similar to AsRef<T>, but with the whole deref coercion magic. Which brings me to the next issue.

Deref coercion is not about dereferencing

If deref coercion was about dereferencing, I'd expect this to work:

use std::ops::Deref;

#[derive(Clone, Copy, Debug)]
struct Foo;

impl Foo {
    fn foo(&self) -> &'static str {
        "I am foo."
    }
}

#[derive(Clone, Copy, Debug)]
struct Bar {
    foo: Foo,
}

impl Deref for Bar {
    type Target = Foo;

    fn deref(&self) -> &Self::Target {
        &self.foo
    }
}

fn print_foo(foo: Foo) {
    println!("{}", foo.foo());
}

fn main() {
    let bar = Bar { foo: Foo };
    print_foo(*&bar);
}

In the line of the call print_foo(*&bar) I'd naively expect the following to happen:

  1. I create a temporary reference to bar, which is of type Bar, that will have type &Bar.
  2. I immediately dereference this reference. Since Bar implements Deref with Target = Foo it should™ dereference to a type Foo.

But no, it doesn't. Because Deref does not actually dereference anything. It just maps reference types. So I'd need to add an extra dereference operator to make the magic happen: print_foo(**&bar); which is equivalent to print_foo(*bar);.
Wait what? bar is not a reference. It is of type Bar, not &Bar. How can I dereference a non-reference type? Well, because due to Deref the code actually desugars to print_foo(*Deref::deref(&bar));. I.e. The Deref trait's method 'deref() gets passed a reference to bar, which it maps to another type of reference, namely &Foo, which then in turn gets dereferenced by the dereference operator *.

On the other hand, the following code works fine:

use std::ops::Deref;

#[derive(Clone, Copy, Debug)]
struct Foo;

impl Foo {
    fn foo(&self) -> &'static str {
        "I am foo."
    }
}

#[derive(Clone, Copy, Debug)]
struct Bar {
    foo: Foo,
}

impl Deref for Bar {
    type Target = Foo;

    fn deref(&self) -> &Self::Target {
        &self.foo
    }
}

fn print_foo(foo: &Foo) {
    println!("{}", foo.foo());
}

fn main() {
    let bar = Bar { foo: Foo };
    print_foo(&bar);
    print_foo(Deref::deref(&bar));
}

There is no dereferencing involved here. We simply take a reference of one type which is automagically ("deref coercion") or explicitly mapped to another reference. Nothing is being dereferenced here.

Where am I going with this?

Probably nowhere. The trait is in standard library for longer than I have been using Rust, so it's not going to go away or be renamed. I rest my case that its name is misleading, as is the term deref coercion. Maybe the official docs can more clearly point out that the trait, despite its name is not about dereferencing and that neither is deref coercion, which rather is reference coercion. Feel free to correct me if I'm wrong with my assertions.

2

It does dereference, in that the output contains one less layer of indirection. For example, if you have &Arc<T> — two indirections — and call deref() on it, you get &T — one indirection. It just happens that the indirection being removed is not the outermost one.

It would certainly be more natural-looking if it returned T, but that would be the job of the hypothetical DerefMove trait. As it is, we have Deref and DerefMut, which each handle one of the cases of the behavior of the place that the implementing type dereferences to.[1]

As I see it, Deref is correctly named because it is what makes the dereference operator work.
Its effect on deref coercion, method dispatch, etc. is just what follows from making the dereference operator work.

  1. By that line of argument, they might better be named PlaceRef and PlaceMut, but that would probably not be any easier to comprehend and use correctly. ↩︎

3

it was very interesting to read, but I think your are bit confused, the unary operator * dereference a type but you are explicit overwriting that, and mapping references.

#[derive(Clone, Copy, Debug)]
struct Foo;

impl Foo {
    fn foo(&self) -> &'static str {
        "I am foo."
    }
}

#[derive(Clone, Copy, Debug)]
struct Bar {
    foo: Foo,
}


fn print_foo(foo: Foo) {
    println!("{}", foo.foo());
}

fn main() {
    let bar = Bar { foo: Foo };
    let refBar = &bar;
    print_foo((*refBar).foo);
}

or like this

use std::ops::Deref;

#[derive(Clone, Copy, Debug)]
struct Foo;

impl Foo {
    fn foo(&self) -> &'static str {
        "I am foo."
    }
}

#[derive(Clone, Copy, Debug)]
struct Bar {
    foo: Foo,
}

impl Deref for Bar {
    type Target = Foo;

    fn deref(&self) -> &Self::Target {
        &self.foo
    }
}

fn print_foo(foo: Foo) {
    println!("{}", foo.foo());
}

fn main() {
    let bar = Bar { foo: Foo };
    print_foo(*(bar.deref()));
}

when you

impl Deref for Bar {
    type Target = Foo;

    fn deref(&self) -> &Self::Target {
        &self.foo
    }
}

you are saying instead of dereference Bar, what makes no sense because it is no a &T, you will do this. All you need to do is *_ to a object type Bar to obtain a Foo, not to a reference, if you want to obtain a reference use deref()

4
Some preliminary observations about the official documentation.

These docs are pretty poor IMO. They're using "deref coercion" as some blanket phrase to cover at least three distinct things:

Their explanation of * is fine, if a bit brief. But that is not deref coercion.

When I explain deref coercion, I emphasize that

  • It takes place "underneath" & or &mut
  • It applies when the target type is known
  • It can insert any number of * applications: &x to &*******x is possible

Method resolution doesn't need a target type, it's a search for a matching method (for some definition of matching). Method resolution applies autoref and does not happen only underneath references, so you may go from x to &x or from &x to x, which deref coercion does not do.

The motivation for Deref implementations that don't reduce indirection is more about method resolution and field access (.) than deref coercion.

The main technical correction related to your intuition I would make is:

Deref is not about changing how * on a &Bar works. It's about defining how * on a Bar works.

The Deref trait is what allows generalizing the * operator from built-in types to any types. Most std implementors are smart pointers of one sort or another, and their implementations do tend to remove a layer of indirection. Using Rc<T> as an example, the implementation defines the * operation on a Rc<T>. It does not change the definition of * on a &Rc<T>.

If the primary motivation is to remove indirection, you may wonder why things work like:

fn deref(&self) -> &Self::Target

*bar ==> *Deref::deref(&bar)
^^^^ the type of that expression is `<_>::Target`

Explanations include

  • We take &self so we don't move owning smart pointers like Box<_> or Rc<_>
  • We can't return Self::Target without moving the target, which we don't want to do
  • We can't return Self::Target when it's not Sized either
  • *bar is a place expression so it can't be the same as just Deref::deref(&bar)

Beyond the technical, I think this is mainly a conversation about being annoyed by a cognitive mismatch.

A lot of your annoyance seems to come from implementations that do not remove indirection, and are more "place conversions" when used with the * operator. But to me that's just a natural consequence of generalizing an operation: whatever the primary motivation, any implementation that matches the Deref signature will do. String + &str concatenates; should we stop calling it Add?

In the generic case, you don't even know how much indirection is or isn't actually being removed. Even in the case of a known foreign type, perhaps tomorrow they'll change their struct to put the target field into a Box to save on inline space. Part of the idea behind traits and generics is you don't have to care.

Assuming you have any interest in changing your perspective, I suggest thinking of Deref as "defining what the * operator does" and as dereferencing in a generic context as "applying the * operator".[2] I think this avoids a lot of the cognitive mismatch.

(There are specific cases where one needs to distinguish Deref based derefencing and built-in dereferencing.)

With the above perspective:

There is no using my_ty + my_ty if you don't impl Add for MyTy, and there is no using *bar if you don't impl Deref for Bar.

And &bar became &*bar, so there was an application of the * operator there.

Deref coercion can certainly be about pointer-chasing dereferencing. But sure, there are also cases that don't remove indirection, but effectively apply an offset... or return some static place even. Again a natural consequence of generalizing an operator / operator overloading.

  1. Also, I would not say T implements Us methods; there is no T::u_method. ↩︎

  2. And similarly for other operator traits. ↩︎

5

Thanks for all your feedback. My core point was to convey, that the Deref trait actually (insert meme) performs reference mapping and that the deref coercion is more of an internal compiler magic rather than a function of the trait. I understand and agree with most counter points you made, though I still have a different understanding of what the term dereferencing means.

6

I think the disagreement here comes from mixing two different meanings of “dereference,” and that’s exactly why the naming is misleading.

If we keep terminology consistent:

  • & = referencing → adds a level of indirection
  • * = dereferencing → removes a level of indirection

That pair is clear and symmetric.

But Deref does not follow that same definition. Its actual behaviour is:

&A → &B

i.e. it maps one reference type to another, rather than removing a reference.

So when you say:

“it removes one layer of indirection”

that only holds if you broaden “indirection” beyond references to include wrapper types like Arc<T>. But that’s already a different abstraction than the & / * pair.

From a naming perspective, this is the inconsistency:

  • * matches the intuitive meaning of dereference (&T → T)
  • Deref does not — it performs reference remapping/projection

So the issue isn’t about how it works but about terminology:

Rust uses “deref” both for actual dereferencing (*) and for reference-to-reference mapping (Deref), which are not the same operation.

That’s why I personally feel misled.

7

This should actually be intuitive given the name of the Defef trait: you implemented Deref for Bar with Target = Foo so now you can use the dereference operator on it to get a Foo!

It starts becoming confusing once you go to see the details of how this is implemented because dereferencing is actually supposed to work on places, and those are not a first class citizen of the language so it needs to be emulated with references.

8

I think there is no broadening going on. Arc<T> is more indirect than T.

It does remove a layer of indirection. Quite literally so. It doesn’t remove a “layer of indirection” that’s due to the “&_” type specifically. But the claim was it “removes a layer of indirection” not “it removes a layer of primitive-reference type”.

Also - depending on your viewpoint - it arguably doesn’t remove the outermost layer of indirection. The argument was that turning &Arc<T> into &T removes a layer of indirection. But the claim was not “it removes the outermost layer of indirection”, anyway :wink:

Of course it is true that there’s nothing forcing you to implement Deref only for cases where an actual level of indirection is removed, even though in many cases implementations do use it for exactly that purpose; perhaps that’s your point?

The same thing of “can be used more generally than what motivated the name originally” applies in other cases, too, of course. E.g. == and the Eq [PartialEq] trait only requires properties of a [partial] equivalence relation, yet the symbolism is rather following a notation for “true”/mathematical equality, and even the documentation) will often switch to use the word “equal” anywhere but in the most technically-correct top-level documentation.

I would question the symmetry here. The dereferencing operator, * works natively with &T or &mut T (and arguably also natively with Box<T>). If you’re looking for symmetry, you’re missing the detail that “undoing” a referencing step with deref will also match up e.g. with the &mut operator in some cases; and can also work on Box<T> without any best “opposite direction” operator that fully cancels out.

Ultimately, this thing just simply isn’t very symmetrical. We have a * operator that’s overloadable; but &-operator can not be overloaded.

9

That's a consequence of language limitation. The intent of the Deref trait is to generalize the * operator by allowing mappings like Box<T> -> T, Arc<T> -> T and so on. So in alternate world it would be defined something like this (not valid Rust):

trait Deref {
    fn deref<T>(place self: Self<T>) -> place T
}

impl Deref for Box {
    fn deref<T>(place self: Box<T>) -> place T { todo!() }
}

But in current Rust:

  1. places are not first class citizens
  2. type constructors are not first class citizens

To resolve #1, we can work behind a reference (still not valid Rust):

trait Deref {
    fn deref<T>(self: &Self<T>) -> &T
}

impl Deref for Box {
    fn deref<T>(self: &Box<T>) -> &T { todo!() }
}

To resolve #2, we apply the trait to type instances themselves:

trait Deref {
    type Target;  // we need this to be able to write the signature of the function below
    fn deref(self: &Self) -> &Self::Target;
}

impl<T> Deref for Box<T> {
    type Target = T;
    fn deref<T>(self: &Box<T>) -> &T { todo!() }
}

which is what we have now. An unfortunate(?) consequence is that we can now also do confusing(?) things like:

impl Deref for Foo {
    type Target = Bar;
    fn deref<T>(self: &Foo) -> &Bar { todo!() }
}

but those do not match the original intent. Just like you can implement e.g. an Add trait by any binary function, not just addition.

After all, even in Rust, not everything is expressed by type. Sometimes you still need to read and follow the documentation to use stuff correctly.

10

A nitpick, only, not relevant to your point about places: if we hypothetically redefined Deref to work on type constructors only, that would make it importantly less powerful. For example, String dereferences to str — no type parameter present. The same pattern applies to CString, OsString, PathBuf, and IoSlice in the standard library.

11

Another thing of note: Dereferencing &T → T in a low-level language like Rust (or the equivalent concepts in C or some others) is arguably a lot less intuitive than one might initially assume.

This is because of the special treatment of what Rust calls “places” and “place expressions” (in C I think it’s called “lvalue”).

When you just write *x for a variable, or *expr() for an expression, that’s usually not corresponding to a direct run-time operation of “you were holding a pointer [i.e. an address] in your hand, then you read from it, and now you are holding the value behind that pointer”. At least if you think of “in your hand” as “in registers”. Let me try to illustrate with an example:

Suppose we have this setup

let var1: u8 = 123;
let var2: &u8 = &var1;
let var3: u8;

and compare something like

// first example
var3 = var1;

vs an alternative operation like

// second example
var3 = *var2;

without any “optimizations” turned on.

This looks like the first example just needs to take that u8 and move it somewhere else; whereas the second example needs to also do “dereferencing”, i.e. take the pointer, dereference it into the u8, put that into var3.

If you look at the assembly, you might notice though, arguably the new sort of instruction that’s being added in the second example is not the part that’s

also do “dereferencing”, i.e. take the pointer, dereference it into the u8

but instead just the

take the pointer

will look new, because - for the machine - “take a pointer, dereference it to u8 (producing a result by value)” is the same kind of operation as the “take that u8” step in the first example:

// first example

        mov     al, byte ptr [rsp + 7]
        mov     byte ptr [rsp + 23], al

// second example

        mov     rax, qword ptr [rsp + 8]
        mov     al, byte ptr [rax]
        mov     byte ptr [rsp + 23], al

(from compiler explorer)

The question (without a 100% clear answer): Which instruction is “added” by the dereferencing?

By means of keeping variables on the stack, and only directly working with values in registers, the thing that appeared extra in this assembly - from using the built-in dereferencing operator, on normal shared references - arguably matches a &&u8 → &u8 operation more closely, doesn’t it?

It could be even clearer if there wasn’t a fused “constant offset from address and load” instruction. Then you could compare something like:
// first example (first instruction split up manually)

; was:  mov     al, byte ptr [rsp + 7]
        lea     rax, [rsp + 7]
        mov     al, byte ptr [rax]
;      ---
        mov     byte ptr [rsp + 23], al

// second example (first instruction split up manually)

; was:  mov    rax, qword ptr [rsp + 8]
        lea     rdi, [rsp + 8]
        mov     rax, qword ptr [rdi]
;      ---
        mov     al, byte ptr [rax]
        mov     byte ptr [rsp + 23], al

arguably the mov rax, qword ptr [rdi] seems to be what’s the extra step now, doesn’t it?

By forcing the compiler to use the actual overloaded Deref impl for the &u8 type, you can actually generate matching assembly that mirrors this pattern (yet inserts the call to Deref::deref, of course:

like first example (just to see things didn't change)

        mov     al, byte ptr [rsp + 7]
        mov     byte ptr [rsp + 23], al

like second example but using `Deref::deref` 

        lea     rdi, [rsp + 8]
        call    qword ptr [rip + <&T as core::ops::deref::Deref>::deref::h7dacbc13b60c7a8b@GOTPCREL]
        mov     al, byte ptr [rax]
        mov     byte ptr [rsp + 23], al

for reference[1] – the deref function being used:

<&T as core::ops::deref::Deref>::deref::h7dacbc13b60c7a8b:
        mov     rax, qword ptr [rdi]
        ret

(also from compiler explorer)

Returning to talking about Rust, not assembly… the dereferencing operator has a lot of different use-cases. You can read from its result (in which case you produce a value), or take a reference to it (for example, something like &*x on an x which is already a reference is basically a no-op. Not just due to optimization – it’s a no-op by definition) or assign to it *x = value; (thought in general that needs DerefMut and only works primitively with mutable references).

This set of options is similar to what you can do with variables and some other language constructs, and it’s called a “place”. A dereference expression *other_expr is thus a place expression.

Interestingly, the argument of * is also treated as a place: For &T that seems redundant, but if you use *foo for some foo: &mut T or foo: Box<T> you wouldn’t want *foo to consume the value foo (e.g. if used in an assignment *foo = bar();).

It’s arguably this special nature that * does inherently turns a place into another place, which makes it correspond so closely in the (unoptimized) assembly to sort-of “doing the dereferencing behind one additional level of indirection” and what motivates the design of Deref trait (in a language like Rust with “by-value” calling conventions) to use a function signature of &T -> &Target.

  1. no pun intended ↩︎

12

Yes, maybe I should have added:

  1. we want to support non-generic pointers too