What are smart pointers?

21

Well, I think I meanwhile have a good feeling regarding when to implement Deref and when not to implement it. But the warning, along with a blurry (or entirely missing) definition of what smart pointers really are, actually adds to my confusion/insecurity rather than avoiding it.

I think the important point for newcomers is that

are three different things. When we implement Deref, this will (among changing the behavior of the dereference operator (*)) enable coercions to the target at several places in our code. And I think idiomatic Rust code should enable these coercions only for a few number of applications ("smart pointers", whatever that is, being one of them).

In particular, the coercions should not be used to implement inheritance and not be used in case of newtypes. (Edit: As far as I understood.) Perhaps it would be better to give negative examples in the documentation of Deref rather than referring to a blurry term with no clear definition? Or provide better positive examples.

22

There is no clear definition. It's a murky intuition formed by generalizing from a few examples. Box, Arc, Rc, Cow are conventionally smart pointers, and guards may be, but they're more importantly "guards." I mean, the fundamental problem here is that there's no useful reason to distinguish between smart pointers and other things except to make a rough analogy with pointers and references for things that are obviously more complex than them.

So you're not going to get a clear answer to that problem, and it's probably the wrong question to be asking in the first place. What data structure you need depends on what problems you're solving, and whether to implement Deref or not is a matter of what tradeoffs you make in your particular use case.

But the warning, along with a blurry (or entirely missing) definition of what smart pointers really are, actually adds to my confusion/insecurity rather than avoiding it.

Some overly technical, over-fit, jargon-laden definition would not be helping you or other newcomers. You should come to that documentation with a set of project requirements in mind, not a bare desire to merely follow some rules. Those requirements will provide valuable context that is needed to interpret the docs. The space of all possible programs is simply far too vast for anyone to outline the circumstances in which any particular design is valuable or not.

4 Likes
23

It should be noted that ManuallyDrop in std::mem - Rust implements Deref and DerefMut even though it is not a smart pointer. Or is it?

24

If that is true, it would be an interesting outcome of this discussion (for me, that is).

I'd like to note that there were some (partially) opposing voices regarding Cow in the previous thread here and here.

I'd agree.

Similarly, even if we would consider a String being a smart pointer, it still is (more importantly) a growable string / buffer.

I think this "rough analogy" can be an important piece when it comes to understanding how certain things are done in Rust. If you're coming from garbage collected language like Java or Python (which some people do), the notion of a data structure being something like a pointer or reference may be not well known. But sooner or later, you'll have to use Arc, Rc, RefMut or the like. And I think it does make sense that the Rust reference mentions these kind of "smart pointers" when it talks about "pointer types".

I think that it's reasonable (and a common practice) to try to describe certain "patterns" in computer programming with a technical term, and I think seeking definitions (or criticizing them) isn't a wrong thing per se.

My original "problem" was that the documentation specifically uses the term to tell me when I should (or rather shouldn't) implement Deref. Questioning that term led me to believe that the warning in the documentation might be misleading, and that's a thing that deserves attention, IMHO. (Maybe I'm the only one seeing a problem here, though.)

That depends. I take warnings seriously. Especially if they are contained in the official documentation and bold. When I see such a warning, I investigate what it means and what I have to take care of, because usually things can go pretty wrong when I don't. When I cannot even investigate or read references or find sources, then… well… I kinda feel like being a bit in a predicament. (By the way: The Wikipedia article on Smart Pointers currently says, "This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources.") One option is to just say, "the warning is imprecise, I'll just ignore it and follow my own guts." Or I ask here in the forum.

Maybe there isn't really an unanimous opinion when and why Deref should be implemented. That'd be an interesting outcome of this discussion as well.

But Deref aside, I still find it of interest to understand what the term "smart pointer" usually means (whether in the context or Rust or generally), because I suspect I will encounter it more often in future.

25

Not sure whether it should or shouldn't be considered a smart pointer. If it is one, we might say it is a "zero-cost smart pointer" that "manages" the pointed-to value by explicitly not managing it on drop.

26

A "smart pointer" is a pointer that's not a "dumb pointer." I.e., it has to do something nontrivial when you operate on it.

But as I hinted at, this isn't useful to anyone. You don't need to know the definition of a car to build one. Just as nobody would ask for the definition of an operating system to know if they've build one. You'll know if you're implementing a smart pointer because nobody has ever done that unintentionally. You implement something that acts like a pointer, and you make it do something interesting.

Nobody will care if you end up saying "according to document A, written by person B, your code C is not a D," because obscure word games and rote classification is one of the more pedantic and useless ways to spend one's time.

What predicament? Do you have an example of a program you were unable to write, or a problem you were unable to solve? As far as I can tell, you're worried about being ignorant about some thing, and that's not a predicament as much as it is the human condition. Nobody is omnipotent. We all get by without knowing some things, but one of the important things you can know is when you're asking a question whose answer doesn't matter.

1 Like
27

I have a different notion of what I need or don't need or what's useful or not when I learn a programming language, but I'll respect your view here.

I don't feel like I need a particular example to justify my wish to understand Rust (or the term "smart pointer") better, but since you asked, I do have an interesting case where I decided to not implement Deref but saw that someone else who wrote something similar did implement Deref:

refid::ById vs by_address::ByAddress (of which the latter implements Deref)

There are also examples from the past, where I (supposedly) wrongly implemented Deref. But that were my first experiments, and I dont think I'd find the code anymore.

Another example is the newtype pattern in general. I would like to know if it is really correct to never implement Deref, or if there are cases when it makes sense. And if so, which cases. But that's just my general interest, I don't currently have a use case here, but I'm sure I'll soon get into the need of implementing a newtype though.

28

It's your choice whether to implement Deref. The main concern here is trying to justify to your users why the added ambiguity and implicitness that results is worth it. "Smart pointer" in this context means that the type's pointer semantics are secondary to the fact that it acts as a storage for some T. In other words, the T being pointed at is far more important to the user than the fact that it's wrapped by something else.

This usually doesn't hold for newtypes, because the newtype pattern is most frequently used to modify the observable behavior of the T inside, rather than just add additional qualifications on a pointer to it. (ByAddress appears to be doing exactly that -- it's a newtype intending to modify the behavior of the pointer, not the pointee.)

But this is just one way to look at it, and it is no less valid than other, conflicting interpretations. SmartPointer<T> isn't a provided abstraction in Rust, so basically the 'canonical' definition of smart pointer in Rust is just 'anything that implements Deref/DerefMut'. But that would be circular reasoning if you're trying to decide whether to implement those traits in accordance with some definition of smart pointer. You'd ultimately have to make that determination based on your project requirements.

1 Like
29

I like to keep thins simple.

A "smart pointer" from its description should be a "pointer" like thing. That is it points at/references something else. That is one should be able to dereference it in order to get at that other thing. Else it is no use as a "pointer". Often a pointer can be changed to point to different instances of the things it can point to, as you do when manipulating linked lists, trees, etc.

"smart" implies it has some other features above just pointing at some othing. What ever they are. Could be it owns the thing and counts how many others are using it so as to know when to drop the thing (Rc). Could be it wraps the thing in some mutual exclusion for thread safety (Arc). Could be many other things besides. *Smart" things.

Is String a smart pointer. Hmm.. perhaps. But the main point of String is to actually be a string. It is a container. So I would say no.

Seems we can implement deref for pretty much anything we like. And have it do whatever we like. Same as we could for the arithmetic operators in many languages. That seems like a bad idea unless ones deref does actually dereference as one would expect from a pointer like thing. Same as it is a bad idea to implement +for types for which arithmetic operators make no sense. One can do it, but it's going to confuse the reader.

There are abstractions here that are at a higher level than the actual language mechanics enforces sometimes. Abstractions that we all need to recognise and follow to avoid confusion.

From this discussion it's clear we can all have somewhat different understandings of what those higher level abstractions/concepts are or include.

1 Like
30

This entire discussion would be moot except that the documentation refers to "smart pointers," so I just filed this as a GitHub issue:

https://github.com/rust-lang/rust/issues/91004

4 Likes
31

Deref's documentation certainly has added to this discussion, and I think the documentation should be fixed in either case.

However, the term "smart pointer" doesn't only appear in the context of Deref. I would even say that it's one of Rust's major peculiarities to utilize "smart pointers" such as

but also other types that might fall into this category such as

Not wanting to imply all of these should be considered smart pointers, but some for sure. No wonder, the term "smart pointer" (or the generalized term "pointer type") also appears in the reference and repeatedly pops up in discussions (which was actually the trigger that made me rise the question).

Being interested in Rust and modern computer programming languages in general, i'm interested in the used terminology, and I don't think this discussion is "moot" after Deref's doc has been fixed. (Besides, the reference uses the terminology as well.)

However, I also understand that this sort of discussion doesn't interest everyone, and that's totally fine.

Having said the above, I think it makes sense to differentiate between the two questions:

  • When and when not to implement Deref
  • What are "smart pointers"

The question when to implement Deref is an important one (and one that may be of more practical value for most people here rather than debating about "academic" terminology). In that context, the ticket created by @2e71828 contains a reference to another discussion (Understanding the perils of Deref) which I haven't read yet. It may also deserve further discussion / documentation / etc., but I meanwhile think that it's best to leave that question out when trying to define what a smart pointer is.

That said, I'm still interested in debating the issue of the terminology "smart pointer" further (for the reasons explained above), but I'm not sure if it's possible to make any progress here. I'll give it a try, however, and throw in some new (updated) hypotheses:

  • Smart pointers are entirely generic in regard to their target, which means String is not a smart pointer.
  • Smart pointers do not always provide a fail-safe (or non-blocking) dereferencing mechanism, which particularly means that Mutex and RefCell but also Weak or Option could be considered smart pointers, and also means that in not all smart pointers can implement Deref in Rust (as Deref::deref should never fail, at least in Rust).
    Note: I'm very unsure on this, and maybe in the context of Rust it makes sense to demand never-failing dereferencing, as it's also the case with ordinary references. Perhaps it makes sense to think about a term like "smart references", which demand fail-safe deref?
  • Smart pointers do not always point to values on the heap.
  • Wrappers which are intended to change the behavior of the wrapped type (newtype pattern) go beyond of what a smart pointer usually does and thus are not considered smart pointers.

I'm not sure if "storage" is the right term. When I think of storage, I think of "wrappers". A Cow::Borrow doesn't store anything. It points to something. Also the Cow::Owned points to something (but adds some "smartness" to dispose of the value when the Cow goes out of scope).

But I do like what you wrote with "the T being pointed at is far more important to the user than the fact that it's wrapped by something else." It's a bit vague though, and it might rule out Mutex and MutexGuard as it's a very important property of these that they actually perform locking.

Interesting fact. Also note that Pin<P> doesn't impose any bounds on P (only in methods / implementations).

Right now this might be an implication of Deref's documentation. But I think that should be changed (for the reasons that came up in this thread). In particular, such a definition would rule out Weak being considered a smart pointer, and it would imply that non-generic types such as String are a smart pointer (which I don't think anymore they are).

32

Maybe we could say:

A String "acts like a smart pointer" to str. But it isn't a smart pointer in the strict sense.

(Just an idea.)

Update:

A comment to ticket #91004 contained some links to the Rust book, which states:

We’ve already encountered a few smart pointers in this book, such as String and Vec<T> in Chapter 8, although we didn’t call them smart pointers at the time. Both these types count as smart pointers because they own some memory and allow you to manipulate it. They also have metadata (such as their capacity) and extra capabilities or guarantees (such as with String ensuring its data will always be valid UTF-8).

So the book considers String and Vec[T] being smart pointers. Here is the link to the page in the book.

33

Maybe the term "smart pointer" just means something a bit different in the context of Rust than it does in other languages like C++. What with being smarter than your average smart pointer.

34

Another thought: Assuming the warning in Deref's documentation ("Deref should only be implemented for smart pointers") was removed (as proposed by the ticket opened, and hopefully replaced by a more helpful hint), then we could simply use the "canonical" or "syntactical" definition:

A smart pointer is any data type that implements Deref.

This would also be consistent with the book, I think, as a String is then indeed a smart pointer to a str. It would also allow the Rust reference to be more precise in regard to what a smart pointer is.

2 Likes
35

I haven't exhaustively examined things but I suspect you're exposing a fuzziness in what people consider the newtype pattern to encompass.

A common purpose of the newtype pattern is to prevent things like DegF(30) + DegC(20) from being equivalent to 30 + 20, which supports a view that coercions are antithetical to the purpose of newtypes.

...but, on the other hand, when you get to something like Path and PathBuf, it gets hazier. Suppose there was a Filename type instead of them just returning OsStr when you call file_name(), as a means to add some extra operations.

The question is whether the fuzzy edge between "OsString derefs to OsStr" and "PathBuf wraps OsString" semantics has a region of overlap and, if it does, do its occupants qualify as being examples of the newtype pattern.

It's entirely possible that it might end up in a tautological "not implementing coercions is part of the definition of a newtype" situation.

1 Like
36

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.