What are smart pointers?

First thing first, as you picked up, my definition of smart pointer is a little sloppy, forgot to say "It needs to manage the caller defined type"

But,I went through your reply and I think what's happening here is that C++ and Rust has a slightly different view on how a smart pointer should behave, and that Rust seems to have a way stricter requirement on what it calls smart pointer.

One of the point you made is that in Rust, for a smart pointer T, it must implement Deref/DerefMut to &U. And by definition of the implementation of Deref/DerefMut, it must also always succeed in coercing to &U. You made a similar note for Option<&U> as well, as it is not able to always be able to deref to &U. However, I do not believe that is a strictly requirement as we can see there are examples in both C++ and Rust that breaks that rule:

  • std::rc/sync::Weak - Weak Pointer does no implement Deref as they are not guarantee to be able to grab the inner U due to the possibility that the last of the Rc/Arc might have already been dropped. But it is widely considered to be smart pointer. This applies to C++ version as well.
  • std::optional - You've noted that Option does not match the Deref behavior in Rust, but std::optional in C++ actually does implement * and -> overload for it. So in essence, std::optional behave way more like a pointer. Of course, we can argue if that is the right thing to do, but what I'm saying is that C++ expect smart pointer to behave like pointer (* const U), while Rust's Deref expect smart pointer to behave like references (&U).

And for a continuation of my "$0.02"

Because "Smart Pointer" ends up describing way too broadly on what it does. It is really mostly describe that you are holding a "Singular" that may have prerequisite/action on the creation of the view (e.g. MutexGuard/RefMut) and/or on the access of the element (e.g. Weak/Cow/Option<&U>) and/or on the end of access (e.g. MutexGuard/RefMut). This is why I said Rust &/&mut behave suspiciously like a smart pointer, as it has all three of those elements (e.g. share XOR mut on create, only &mut access &mut functions and new &mut only after all share/mut access dropped).

Sure, you can restrict the definition to what Rust had described, but what you really ended up is defining that T implementing Deref/DerefMut to the specification and semantic.
At this point, giving it the name "Smart Pointer" is not very useful.
And like I mentioned before, those rules you gave ended up classifying both MutexGuard and Box in the same category. One which I would actually treat more like &U, while the other I would treat more like a structure member.

Regarding "caller defined type", if I understand it right, then this would exclude String, CString, OsString, and PathBuf from being smart pointers. I think that is a valid way to see things, but I would like to note that then the warning in Deref's documentation would be incorrect as I think it's reasonable that String does implement Deref for str. (And it would be same reasonable for a couple of other types that manage a specific, non-generic inner type.)

Moreover, pushing this further, one could claim that Vec<T> isn't a smart pointer either. Even if it is generic over T, it is not entirely generic in regard to the Deref::Target (as Deref::Target is always a slice and never any non-slice). Thus, it always manages slices (and nothing else), while an Arc, which we all agree is a smart pointer, is completely generic in regard of what it manages.

Not sure what "managing" means though. If the managed type we're talking about is generic, then the management will always be very… well… generic :grin:. On the other hand, Strings or Vecs aren't totally generic on their target, which means they can offer more functionality as they know what they manage (str and slice).

I don't want to say that String is or isn't a smart pointer to str, but I would like to say that there are valid arguments for either claiming it is or it isn't one. In the latter case, of course my attempt for a definition is flawed (at least assuming that it's indeed reasonable to implement Deref<Target=str> for String) and the documentation in Deref should be fixed.

I think it can be natural to expect stricter requirements in Rust than in C++. In Rust, references are always guaranteed to be valid, while pointers in C(++) are not. Thus it would be not surprising if we expect any other pointer type (with the exception of raw pointers that are used in unsafe Rust) to also be always valid, i.e. Deref::deref must never fail.

It is interesting that std::optional in C++ overloads * and ->, but given that in C++ pointers aren't guaranteed to be valid, there seems no problem there. So maybe when being in the Rust world, we should rather speak of "smart references" :grinning_face_with_smiling_eyes: (not entirely serious here, but perhaps that clarifies what I mean).

Are you sure it's about being "singular" and not about being "generic"? (Note I do not consider Vec<T> to be entirely generic in regard to smart pointing, as Deref::Target is always a slice and not just any type).

Yes, that's what I meant with:

I agree it would be good to have a better definition.

I don't understand. Can you explain/rephrase what you mean?

One more thing to consider: If we relax the rule on the access always being successful, we might also consider Mutex<T> to be a smart pointer to T. Isn't Mutex quite similar to Weak in that matter (managing some pointed-to generic value that might or might not be accessible)?

I opened this thread with a few questions, and I feel like the questions don't really have clear answers, and on top of that, I have even more questions:

  • Does a smart pointer need to be generic in regard to the managed "target" (in case it's implementing Deref, that means Deref::Target is just a generic type U, e.g. as passed as a type parameter to Arc<U>, Rc<U>, MutexGuard<U>, etc.)
  • Does a smart pointer always need to provide a non-failing dereferencing mechanism? (If not, then maybe Mutex<U> is also a smart pointer?)
  • Is it reasonable to implement Deref for non-smart-pointers? If yes, under which circumstances?
  • The reference mentions "smart pointers" in the "pointer types" subsection. Should the reference give any definition or is that part of the reference non-normative anyway? (In the second case, maybe a short note could be added to that paragraph?)

And most importantly… I still have no clear answer on the main question: "What are smart pointers?"

This is important as long as the Deref documentation warns us to not implement Deref for any non-smart-pointers. So maybe that documentation should be adjusted/improved?

I think the warning is more advice rather than an absolute rule. It's to "avoid confusion". You could probably cause confusion by overloading other operators in evil ways, evil that is to anyone trying to understand the program. Macros can probably also be used to make code incomprehensible. Do we have any examples of "questionable" use of Deref? I think the discussion is a bit academic, although I have not followed it all that closely, so apologies if I am missing the point.

Well, I think I meanwhile have a good feeling regarding when to implement Deref and when not to implement it. But the warning, along with a blurry (or entirely missing) definition of what smart pointers really are, actually adds to my confusion/insecurity rather than avoiding it.

I think the important point for newcomers is that

are three different things. When we implement Deref, this will (among changing the behavior of the dereference operator (*)) enable coercions to the target at several places in our code. And I think idiomatic Rust code should enable these coercions only for a few number of applications ("smart pointers", whatever that is, being one of them).

In particular, the coercions should not be used to implement inheritance and not be used in case of newtypes. (Edit: As far as I understood.) Perhaps it would be better to give negative examples in the documentation of Deref rather than referring to a blurry term with no clear definition? Or provide better positive examples.

There is no clear definition. It's a murky intuition formed by generalizing from a few examples. Box, Arc, Rc, Cow are conventionally smart pointers, and guards may be, but they're more importantly "guards." I mean, the fundamental problem here is that there's no useful reason to distinguish between smart pointers and other things except to make a rough analogy with pointers and references for things that are obviously more complex than them.

So you're not going to get a clear answer to that problem, and it's probably the wrong question to be asking in the first place. What data structure you need depends on what problems you're solving, and whether to implement Deref or not is a matter of what tradeoffs you make in your particular use case.

But the warning, along with a blurry (or entirely missing) definition of what smart pointers really are, actually adds to my confusion/insecurity rather than avoiding it.

Some overly technical, over-fit, jargon-laden definition would not be helping you or other newcomers. You should come to that documentation with a set of project requirements in mind, not a bare desire to merely follow some rules. Those requirements will provide valuable context that is needed to interpret the docs. The space of all possible programs is simply far too vast for anyone to outline the circumstances in which any particular design is valuable or not.

4 Likes

It should be noted that ManuallyDrop in std::mem - Rust implements Deref and DerefMut even though it is not a smart pointer. Or is it?

If that is true, it would be an interesting outcome of this discussion (for me, that is).

I'd like to note that there were some (partially) opposing voices regarding Cow in the previous thread here and here.

I'd agree.

Similarly, even if we would consider a String being a smart pointer, it still is (more importantly) a growable string / buffer.

I think this "rough analogy" can be an important piece when it comes to understanding how certain things are done in Rust. If you're coming from garbage collected language like Java or Python (which some people do), the notion of a data structure being something like a pointer or reference may be not well known. But sooner or later, you'll have to use Arc, Rc, RefMut or the like. And I think it does make sense that the Rust reference mentions these kind of "smart pointers" when it talks about "pointer types".

I think that it's reasonable (and a common practice) to try to describe certain "patterns" in computer programming with a technical term, and I think seeking definitions (or criticizing them) isn't a wrong thing per se.

My original "problem" was that the documentation specifically uses the term to tell me when I should (or rather shouldn't) implement Deref. Questioning that term led me to believe that the warning in the documentation might be misleading, and that's a thing that deserves attention, IMHO. (Maybe I'm the only one seeing a problem here, though.)

That depends. I take warnings seriously. Especially if they are contained in the official documentation and bold. When I see such a warning, I investigate what it means and what I have to take care of, because usually things can go pretty wrong when I don't. When I cannot even investigate or read references or find sources, then… well… I kinda feel like being a bit in a predicament. (By the way: The Wikipedia article on Smart Pointers currently says, "This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources.") One option is to just say, "the warning is imprecise, I'll just ignore it and follow my own guts." Or I ask here in the forum.

Maybe there isn't really an unanimous opinion when and why Deref should be implemented. That'd be an interesting outcome of this discussion as well.

But Deref aside, I still find it of interest to understand what the term "smart pointer" usually means (whether in the context or Rust or generally), because I suspect I will encounter it more often in future.

Not sure whether it should or shouldn't be considered a smart pointer. If it is one, we might say it is a "zero-cost smart pointer" that "manages" the pointed-to value by explicitly not managing it on drop.

A "smart pointer" is a pointer that's not a "dumb pointer." I.e., it has to do something nontrivial when you operate on it.

But as I hinted at, this isn't useful to anyone. You don't need to know the definition of a car to build one. Just as nobody would ask for the definition of an operating system to know if they've build one. You'll know if you're implementing a smart pointer because nobody has ever done that unintentionally. You implement something that acts like a pointer, and you make it do something interesting.

Nobody will care if you end up saying "according to document A, written by person B, your code C is not a D," because obscure word games and rote classification is one of the more pedantic and useless ways to spend one's time.

What predicament? Do you have an example of a program you were unable to write, or a problem you were unable to solve? As far as I can tell, you're worried about being ignorant about some thing, and that's not a predicament as much as it is the human condition. Nobody is omnipotent. We all get by without knowing some things, but one of the important things you can know is when you're asking a question whose answer doesn't matter.

1 Like

I have a different notion of what I need or don't need or what's useful or not when I learn a programming language, but I'll respect your view here.

I don't feel like I need a particular example to justify my wish to understand Rust (or the term "smart pointer") better, but since you asked, I do have an interesting case where I decided to not implement Deref but saw that someone else who wrote something similar did implement Deref:

refid::ById vs by_address::ByAddress (of which the latter implements Deref)

There are also examples from the past, where I (supposedly) wrongly implemented Deref. But that were my first experiments, and I dont think I'd find the code anymore.

Another example is the newtype pattern in general. I would like to know if it is really correct to never implement Deref, or if there are cases when it makes sense. And if so, which cases. But that's just my general interest, I don't currently have a use case here, but I'm sure I'll soon get into the need of implementing a newtype though.

It's your choice whether to implement Deref. The main concern here is trying to justify to your users why the added ambiguity and implicitness that results is worth it. "Smart pointer" in this context means that the type's pointer semantics are secondary to the fact that it acts as a storage for some T. In other words, the T being pointed at is far more important to the user than the fact that it's wrapped by something else.

This usually doesn't hold for newtypes, because the newtype pattern is most frequently used to modify the observable behavior of the T inside, rather than just add additional qualifications on a pointer to it. (ByAddress appears to be doing exactly that -- it's a newtype intending to modify the behavior of the pointer, not the pointee.)

But this is just one way to look at it, and it is no less valid than other, conflicting interpretations. SmartPointer<T> isn't a provided abstraction in Rust, so basically the 'canonical' definition of smart pointer in Rust is just 'anything that implements Deref/DerefMut'. But that would be circular reasoning if you're trying to decide whether to implement those traits in accordance with some definition of smart pointer. You'd ultimately have to make that determination based on your project requirements.

1 Like

I like to keep thins simple.

A "smart pointer" from its description should be a "pointer" like thing. That is it points at/references something else. That is one should be able to dereference it in order to get at that other thing. Else it is no use as a "pointer". Often a pointer can be changed to point to different instances of the things it can point to, as you do when manipulating linked lists, trees, etc.

"smart" implies it has some other features above just pointing at some othing. What ever they are. Could be it owns the thing and counts how many others are using it so as to know when to drop the thing (Rc). Could be it wraps the thing in some mutual exclusion for thread safety (Arc). Could be many other things besides. *Smart" things.

Is String a smart pointer. Hmm.. perhaps. But the main point of String is to actually be a string. It is a container. So I would say no.

Seems we can implement deref for pretty much anything we like. And have it do whatever we like. Same as we could for the arithmetic operators in many languages. That seems like a bad idea unless ones deref does actually dereference as one would expect from a pointer like thing. Same as it is a bad idea to implement +for types for which arithmetic operators make no sense. One can do it, but it's going to confuse the reader.

There are abstractions here that are at a higher level than the actual language mechanics enforces sometimes. Abstractions that we all need to recognise and follow to avoid confusion.

From this discussion it's clear we can all have somewhat different understandings of what those higher level abstractions/concepts are or include.

1 Like

This entire discussion would be moot except that the documentation refers to "smart pointers," so I just filed this as a GitHub issue:

https://github.com/rust-lang/rust/issues/91004

4 Likes

Deref's documentation certainly has added to this discussion, and I think the documentation should be fixed in either case.

However, the term "smart pointer" doesn't only appear in the context of Deref. I would even say that it's one of Rust's major peculiarities to utilize "smart pointers" such as

but also other types that might fall into this category such as

Not wanting to imply all of these should be considered smart pointers, but some for sure. No wonder, the term "smart pointer" (or the generalized term "pointer type") also appears in the reference and repeatedly pops up in discussions (which was actually the trigger that made me rise the question).

Being interested in Rust and modern computer programming languages in general, i'm interested in the used terminology, and I don't think this discussion is "moot" after Deref's doc has been fixed. (Besides, the reference uses the terminology as well.)

However, I also understand that this sort of discussion doesn't interest everyone, and that's totally fine.

Having said the above, I think it makes sense to differentiate between the two questions:

  • When and when not to implement Deref
  • What are "smart pointers"

The question when to implement Deref is an important one (and one that may be of more practical value for most people here rather than debating about "academic" terminology). In that context, the ticket created by @2e71828 contains a reference to another discussion (Understanding the perils of Deref) which I haven't read yet. It may also deserve further discussion / documentation / etc., but I meanwhile think that it's best to leave that question out when trying to define what a smart pointer is.

That said, I'm still interested in debating the issue of the terminology "smart pointer" further (for the reasons explained above), but I'm not sure if it's possible to make any progress here. I'll give it a try, however, and throw in some new (updated) hypotheses:

  • Smart pointers are entirely generic in regard to their target, which means String is not a smart pointer.
  • Smart pointers do not always provide a fail-safe (or non-blocking) dereferencing mechanism, which particularly means that Mutex and RefCell but also Weak or Option could be considered smart pointers, and also means that in not all smart pointers can implement Deref in Rust (as Deref::deref should never fail, at least in Rust).
    Note: I'm very unsure on this, and maybe in the context of Rust it makes sense to demand never-failing dereferencing, as it's also the case with ordinary references. Perhaps it makes sense to think about a term like "smart references", which demand fail-safe deref?
  • Smart pointers do not always point to values on the heap.
  • Wrappers which are intended to change the behavior of the wrapped type (newtype pattern) go beyond of what a smart pointer usually does and thus are not considered smart pointers.

I'm not sure if "storage" is the right term. When I think of storage, I think of "wrappers". A Cow::Borrow doesn't store anything. It points to something. Also the Cow::Owned points to something (but adds some "smartness" to dispose of the value when the Cow goes out of scope).

But I do like what you wrote with "the T being pointed at is far more important to the user than the fact that it's wrapped by something else." It's a bit vague though, and it might rule out Mutex and MutexGuard as it's a very important property of these that they actually perform locking.

Interesting fact. Also note that Pin<P> doesn't impose any bounds on P (only in methods / implementations).

Right now this might be an implication of Deref's documentation. But I think that should be changed (for the reasons that came up in this thread). In particular, such a definition would rule out Weak being considered a smart pointer, and it would imply that non-generic types such as String are a smart pointer (which I don't think anymore they are).

Maybe we could say:

A String "acts like a smart pointer" to str. But it isn't a smart pointer in the strict sense.

(Just an idea.)


Update:

A comment to ticket #91004 contained some links to the Rust book, which states:

We’ve already encountered a few smart pointers in this book, such as String and Vec<T> in Chapter 8, although we didn’t call them smart pointers at the time. Both these types count as smart pointers because they own some memory and allow you to manipulate it. They also have metadata (such as their capacity) and extra capabilities or guarantees (such as with String ensuring its data will always be valid UTF-8).

So the book considers String and Vec[T] being smart pointers. Here is the link to the page in the book.

Maybe the term "smart pointer" just means something a bit different in the context of Rust than it does in other languages like C++. What with being smarter than your average smart pointer.

Another thought: Assuming the warning in Deref's documentation ("Deref should only be implemented for smart pointers") was removed (as proposed by the ticket opened, and hopefully replaced by a more helpful hint), then we could simply use the "canonical" or "syntactical" definition:

A smart pointer is any data type that implements Deref.

This would also be consistent with the book, I think, as a String is then indeed a smart pointer to a str. It would also allow the Rust reference to be more precise in regard to what a smart pointer is.

2 Likes

I haven't exhaustively examined things but I suspect you're exposing a fuzziness in what people consider the newtype pattern to encompass.

A common purpose of the newtype pattern is to prevent things like DegF(30) + DegC(20) from being equivalent to 30 + 20, which supports a view that coercions are antithetical to the purpose of newtypes.

...but, on the other hand, when you get to something like Path and PathBuf, it gets hazier. Suppose there was a Filename type instead of them just returning OsStr when you call file_name(), as a means to add some extra operations.

The question is whether the fuzzy edge between "OsString derefs to OsStr" and "PathBuf wraps OsString" semantics has a region of overlap and, if it does, do its occupants qualify as being examples of the newtype pattern.

It's entirely possible that it might end up in a tautological "not implementing coercions is part of the definition of a newtype" situation.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.