Following @Cyborus04's advice, I'd like to start a new thread for the topic of "What is a smart pointer?" (If anyone is interested in discussing this yet.)
I think @trentj brought up some interesting thoughts:
Some questions remain:
Is a "smart pointer" a "pointer"? (I guess that depends on preference of word usage and context.)
Are there really data types implementing Deref that are not smart pointers but have a good reason to implement Deref? (Note that this would violate the recommendation in the documentation of Deref.)
Any responses are welcome, even speculations, personal opinion or preferences, or references to documentation or literature. But if you do, please keep the Code of Conduct in mind.
(Not sure if there's any interest in discussing this further though.)
String, and more generally, Vec<T> could indeed qualify as very wide smart pointers, with the "middle-wide" versions of themselves being Box<str> and Box<[T]> respectively:
they do feature indirection;
implement Deref;
are "smart": they feature drop glue as well as grow-capabilities.
It will only feature guaranteed indirection when the owned variant does: a counter-example would be Cow::Owned(42). It's a Cow<'_, i32> with no indirection inside it.
The previous Cow would be an example.
Too many double quotes here to fully know what the question really is. If we qualified a "pointer" as something featuring indirection, then I'd say that yes, being "a pointer" is a necessary requirement for being a "smart" one, if we also assume that a smart pointer needs to feature, w.r.t.Deref, indirection between Self and Self::Output (which was among the three bullets, and seems to be a sensible definition).
I wouldn't, however, say that an impl Deref is necessarily a pointer. A fortiori, not a smart pointer either (c.f., Cow<i32>).
Contra: Pointers personally make me think instantly on C, where pointers are just a memory address. Note that in the Java world, for example, I mostly hear the term "reference" instead of "pointer".
On the other hand, we have wide-pointers, which also contain more than the memory address. See also the experimental API std::ptr::Pointee in that matter.
I have to admit though, that a String has additional functionality. It is what other languages sometimes call "string buffer". So I guess being a smart pointer is just one aspect here.
On the other hand, each smart pointer comes with some extended functionality. But here, the functionality is very specific to the contained type. String even has an own method String::len (which you can see is implemented separately from str::len if you look into the source).
To reason my statement:
I think semantics matter here. Of course Cow::Owned will let the value livestore the value wherever the Cow lives, but semantically, a Cow<'_, T> points/refers/redirects to a value of type T (that may be on the stack if the Cow is on the stack, but does that matter?). Cow isn't even supposed to add functionality here. It also isn't a newtype because we don't want to deliberately create a different type here to achieve different behavior.
Smart pointers don't have a very precise meaning so it's more of a "I know it when I see it" thing.
In general, I would say a smart pointer...
Wraps some other value
Implements Deref and DerefMut (where applicable) to give you a reference to that value, with dereferencing being a cheap operation which returns the same reference every time
Manages the wrapped value in some way (e.g. Box manages memory, RefCell's Ref manages sharing xor mutability)
Almost all uses of the smart pointer go through the Deref implementation, with next to no method calls which take the smart pointer itself as a receiver
Using that definition, String and Cow would be considered smart pointers, but something like RefCell or AtomicPtr wouldn't.
This concept is orthogonal to wide/fat pointers (i.e. pointer + metadata).
There is some sense in which Arc<T> is definitely a smart pointer in a way that a Vec<T> isn't. It probably comes down to all the extra features Vec has. I definintely don't think of Vec as a smart manager for a slice, even if that's not technically wrong either.
I think different authors have written it differently - some of the books have called Vec<T> a smart pointer, right? But I don't personally understand why one would say it like that - maybe it's meant to be draw a parallel, help explain the language?
By my definition they would because they mediate sharing xor mutability.
Yeah, that's where the line blurs. A lot of Vec's usefulness comes from dereferencing to a slice, but at the same time you've got things like push() and drain() and retain() which break that mould.
Once people get a better understanding of Rust, it's probably worth saying that being a "smart pointer" isn't a binary thing and that you can have varying degrees of "smart pointer-iness". The phrase, "smart pointer", is just shorthand for some common conventions related to implementing Deref because it makes it easier to discuss or teach.
Any explanation of an observed phenomenon which, while not 100% scientifically accurate, is simple enough, and just accurate enough, to convey the beginnings of understanding to anyone who is new to the subject. There is always time to fill them in on the fine detail further down the road.
Regarding Cow being or not being a smart point pointer, how about the following (made up) type Finalize<T>, which could be used to prohibit mutation of an owned value:
mod finalize {
pub struct Finalized<T>(T);
impl<T> Finalized<T> {
pub fn new(inner: T) -> Self {
Finalized(inner)
}
}
impl<T> std::ops::Deref for Finalized<T> {
type Target = T;
fn deref(&self) -> &Self::Target {
&self.0
}
}
}
fn main() {
use self::finalize::*;
let mut buf = String::new();
buf.push_str("Hello");
buf.push_str(" World!");
let s: Finalized<String> = Finalized::new(buf);
println!("s has capacity = {}", s.capacity());
// But we can't modify `s` anymore:
// let Finalized(inner) = s;
}
I guess that's quite similar to Pin. Neither Cow::Owned nor Finalized store the value elsewhere (e.g. on the heap). I would still say Box<T>, Cow<T>, Pin<T>, and Finalized<T> are all smart pointers to T.
To give a different yet similar example which I would not consider to be a smart pointer (and I believe everyone will agree at least here), consider the following type CaseInsensitive, which is a newtype:
Both Finalized<String> and CaseInsensitive store a String (and nothing else). Yet I'd call the first one a smart pointer, and the second not.
Finalized<String> is really meant to (immutably) point to a value of type String and to allow – through manual or automatic dereference – operations that this type (String) offers (and no different behavior).
There's an important difference between Finalize and Pin's Deref implementations:
impl<T> std::ops::Deref for Finalized<T> {
type Target = T;
impl<P: Deref> Deref for Pin<P> {
type Target = P::Target;
// ^^^^^^^^^
that is, Pin<P> is a pointer because P itself is (presumed to be) a pointer, and Pin<P> implements Deref by delegating to P, but Finalized<T> implements Deref just by returning an internal reference.
You are right, Deref::Target is T in case of Finalize<T> (which holds an inner T), and Deref::Target is <P as Deref>::Target in case of Pin<P> (which holds an inner P).
Thus Pin and Finalize aren't as similar as I thought.
It actually doesn't matter that the stored type (in .0) is the same type we point to. It's more a coincidence here.
What matters is that Finalized<String> is meant to (immutably) point to a value of a certain type (which happens to also be String) and to allow – through manual or automatic dereference – operations which that (second) type offers (and no different behavior).
I think that "Smart Pointer" is quite an archaic description that is mostly used in the context of C++, and really is more used to describe an implementation detail.
From what I can see, a smart pointer in C++ is a type that is:
A struct/class that is generic to a caller defined type
Contain a destructor
Which means from a design point of view, that description is a little to general. That description would include both MutexGuard and Box/Unique_Ptr in the same category, two very different types with two very different use cases. This also behaves a little weird in Rust because &T/&mut T dropping out of scope can actually change what is valid code, which raise the question if &T/&mut T are "Smart" pointers
What I think a better distinction for describing a caller defined generic type are these two attributes:
View/Container: the distinction here is that a Container is where the destruction of T is tied to the destruction of the Container, while a View would not. This is where PhantomData would show up.
Singular/Multi: the distinction here is if the type holds a single instance of T, or multiple instance of T.
With those two classification, we can split up most caller defined generic types in this matrix
I find these distinction more useful in analyzing code, particularly in function definition. It also describes how we want to use the object, and separate out from how it would be implemented. So some observation:
It explains why in C++, passing std::unique_ptr around as function parameter by default is a bad idea, as you are literally passing around a container, despite that type's name of "_ptr". What people mostly want is a std::unique_view, which the closest relative in C++ is &T/*T. We are lucky in Rust because we have &mut T.
Also shows why we want to use Option<&T>/Views more often when passing parameters into a function if we don't want to give ownership, but return types can more readily be Containers.
We can see that function that gives out Containers type really a general case of Constructor/Factory function.
Container type can be easily/cheaply decayed into their View counterpart, but not the other way around
Singular and Multi types can quite often be cheaply converted between each other. However, only Singular to Mutli type is a lossless conversion, but Multi to Singular type only allow a subset of values to be shown.
Because of the relationship described, its mean in the generality chart, a function that accepts a Multi View is the most general type of function, as it pretty much accept all types can be converted to it. It is also the least powerful as it can make way fewer assumption.
And just to note, this classify Cow as a Singular Container, which has a very cheap and well defined conversion to a Singular View type of the contained value.
I always thought of "smart pointers" as some wrapper whose function it is to add features to another pointer/object while transparently proxy:ing (acting as) the inner object [implied: without making any attempts to hide or change the behavior of the inner object].
This thread has taught me that if I didn't learn something from a strict definition, then I shouldn't assume there is one.
We had similar differing p.o.v.'s regarding Rust, where you could describe a smart pointer
formally as "implements Deref" (in C++ that'd be overloading * and ->, I guess?),
or by talking about what the type contains, which other properties/behavior it shows (e.g. uses "indirection" (whatever that actually is) or has some drop glue / destructor, etc).
I don't want to compare this 1:1, but I observe that both in C++ and in Rust, there seems to be a "formal" view to say what is a smart pointer:
In C++: overloads * and ->
In Rust: implements Deref (or DerefMut in case of mutability)
And we have other views that reason by semantics or other properties whether we have a smart pointer or not.
Even if the term "smart pointer" originates in the C++ world, we can't deny that it exists in the Rust world:
The documentation of the standard library also refers to the term "smart pointer" by warning that Deref should only be implemented for "smart pointers".
Regarding 1:
Smart pointers
The standard library contains additional 'smart pointer' types beyond references and raw pointers.
Regarding 2:
On the other hand, the rules regarding Deref and DerefMut were designed specifically to accommodate smart pointers. Because of this, Deref should only be implemented for smart pointers to avoid confusion.
Now if we would define smart pointers in a formal way, then we would have a problem with the documentation of Deref as the warning notice is not really helpful: We should only implement Deref for smart pointers, but anything that implements Deref is a smart pointer, so everything is fine always .
Thus for me as a "user" of Rust, I need to understand when to implement Deref and when not. Currently, the documentation ties this to the phrase "smart pointer", thus I see a need to discuss the definition of what is a "smart pointer" in a Rust context, ideally not by defining it through "implements Deref" as that leads to a cycle.
I think it's reasonable to look at
which types implement Deref in order to get a clue what's really special about them, and
the consequences of implementing Deref (mostly automatic coercion, I think).
I would also like to note the following important note in the Deref documentation:
[…], [the] trait [Deref] should never fail. Failure during dereferencing can be extremely confusing when Deref is invoked implicitly.
Both Option<&T> and MutexGuard are in the "Singular" and "View" categories.
While I think it's an interesting classification, I'd like to point out that Option<&T> is different from MutexGuard (which implements Deref) in that way that &Option<&T> cannot be converted to &T without failure, while &MutexGuard<T> can. That is an important difference and the reason why Option<&T> must not implement Deref<Target=T>.
This leads me to the following attempt to define what a "smart pointer" is, in the Rust world:
A type T is a smart pointer to U if it fulfills all of the following properties:
T is neither a raw pointer to U nor a reference to U.
There exists a conversion from &T to &U which should never fail.
Due to semantics, automatic coercions from &T to &U are always wanted at coercion sites.
Admittingly, rule 3 is a bit fuzzy yet. But what does these rules say?
Rule 1 basically distinguishes smart pointers from other "pointer types" in Rust. In this context, it is also interesting what @Sykout said here:
I would say there is really a strong similarity between smart pointers and references. I would say the only reason why (Rust) references aren't smart pointers is a matter of definition (hence rule 1).
Rule 2 basically says that if we get/have any T by reference (i.e. we actually have &T), we can always get a (normal) reference to U (i.e. we can convert to &U). In other words: T points to U, because we can follow (i.e. dereference) a T (we only need a reference to T for that) to get to an U (with immutable access only, i.e. we get a &U).
Rule 3 is most tricky, because it's quite blurry yet. When do semantics really want us to enable automatic coercion? This rule again is a bit cyclic as it basically says that it's a smart pointer if we want to implement Deref on it. If I remember some other threads correctly, most people said the newtype pattern should not use Deref and thus do not enable automatic coercion. The same holds for something like inheritance, where Deref and/or automatic coercions are a bad idea. (I think it's more obvious on the latter case, and maybe there is no 100% consensus on all newtypes? Or is there?)
Let's look at some examples:
References fulfill rule 2 (we can convert &&T to &T) and rule 3 (we can pass a &&T where a &T is expected), but they fail rule 1.
Option<&T> fulfills rule 1 (it is neither a raw pointer nor a reference), but fails rule 2 (we cannot convert from &Option<&T> to &T without panicking in the None case). Thus also rule 3 is failed.
Mutex<T> fails rule 2.
MutexGuard<T> certainly fulfills rule 1 and 2. I guess whether rule 3 is fulfilled is a matter of preference (due to how blurry that rule is yet).
Box<T> fulfills all three rules.
Finalized<T> (as defined above) definitely fulfills all three rules.
Cow<T> also fulfills all three rules.
Pin<T> with <T as Deref>::Target being the type "pointed to" also fulfills all three rules.
CaseInsensitive (as also defined above) is the newtype case, where it's usually said that we do not want automatic coercion (thus failing rule 3).
It would be nice to phrase rule 3 in a better way than this:
Due to semantics, automatic coercions from &T to &U are always wanted at coercion sites.
First thing first, as you picked up, my definition of smart pointer is a little sloppy, forgot to say "It needs to manage the caller defined type"
But,I went through your reply and I think what's happening here is that C++ and Rust has a slightly different view on how a smart pointer should behave, and that Rust seems to have a way stricter requirement on what it calls smart pointer.
One of the point you made is that in Rust, for a smart pointer T, it must implement Deref/DerefMut to &U. And by definition of the implementation of Deref/DerefMut, it must also always succeed in coercing to &U. You made a similar note for Option<&U> as well, as it is not able to always be able to deref to &U. However, I do not believe that is a strictly requirement as we can see there are examples in both C++ and Rust that breaks that rule:
std::rc/sync::Weak - Weak Pointer does no implement Deref as they are not guarantee to be able to grab the inner U due to the possibility that the last of the Rc/Arc might have already been dropped. But it is widely considered to be smart pointer. This applies to C++ version as well.
std::optional - You've noted that Option does not match the Deref behavior in Rust, but std::optional in C++ actually does implement * and -> overload for it. So in essence, std::optional behave way more like a pointer. Of course, we can argue if that is the right thing to do, but what I'm saying is that C++ expect smart pointer to behave like pointer (* const U), while Rust's Deref expect smart pointer to behave like references (&U).
And for a continuation of my "$0.02"
Because "Smart Pointer" ends up describing way too broadly on what it does. It is really mostly describe that you are holding a "Singular" that may have prerequisite/action on the creation of the view (e.g. MutexGuard/RefMut) and/or on the access of the element (e.g. Weak/Cow/Option<&U>) and/or on the end of access (e.g. MutexGuard/RefMut). This is why I said Rust &/&mut behave suspiciously like a smart pointer, as it has all three of those elements (e.g. share XOR mut on create, only &mut access &mut functions and new &mut only after all share/mut access dropped).
Sure, you can restrict the definition to what Rust had described, but what you really ended up is defining that T implementing Deref/DerefMut to the specification and semantic.
At this point, giving it the name "Smart Pointer" is not very useful.
And like I mentioned before, those rules you gave ended up classifying both MutexGuard and Box in the same category. One which I would actually treat more like &U, while the other I would treat more like a structure member.
Regarding "caller defined type", if I understand it right, then this would exclude String, CString, OsString, and PathBuf from being smart pointers. I think that is a valid way to see things, but I would like to note that then the warning in Deref's documentation would be incorrect as I think it's reasonable that String does implement Deref for str. (And it would be same reasonable for a couple of other types that manage a specific, non-generic inner type.)
Moreover, pushing this further, one could claim that Vec<T> isn't a smart pointer either. Even if it is generic over T, it is not entirely generic in regard to the Deref::Target (as Deref::Target is always a slice and never any non-slice). Thus, it always manages slices (and nothing else), while an Arc, which we all agree is a smart pointer, is completely generic in regard of what it manages.
Not sure what "managing" means though. If the managed type we're talking about is generic, then the management will always be very… well… generic . On the other hand, Strings or Vecs aren't totally generic on their target, which means they can offer more functionality as they know what they manage (str and slice).
I don't want to say that String is or isn't a smart pointer to str, but I would like to say that there are valid arguments for either claiming it is or it isn't one. In the latter case, of course my attempt for a definition is flawed (at least assuming that it's indeed reasonable to implement Deref<Target=str> for String) and the documentation in Deref should be fixed.
I think it can be natural to expect stricter requirements in Rust than in C++. In Rust, references are always guaranteed to be valid, while pointers in C(++) are not. Thus it would be not surprising if we expect any other pointer type (with the exception of raw pointers that are used in unsafe Rust) to also be always valid, i.e. Deref::deref must never fail.
It is interesting that std::optional in C++ overloads * and ->, but given that in C++ pointers aren't guaranteed to be valid, there seems no problem there. So maybe when being in the Rust world, we should rather speak of "smart references" (not entirely serious here, but perhaps that clarifies what I mean).
Are you sure it's about being "singular" and not about being "generic"? (Note I do not consider Vec<T> to be entirely generic in regard to smart pointing, as Deref::Target is always a slice and not just any type).
Yes, that's what I meant with:
I agree it would be good to have a better definition.
I don't understand. Can you explain/rephrase what you mean?
One more thing to consider: If we relax the rule on the access always being successful, we might also consider Mutex<T> to be a smart pointer to T. Isn't Mutex quite similar to Weak in that matter (managing some pointed-to generic value that might or might not be accessible)?
I opened this thread with a few questions, and I feel like the questions don't really have clear answers, and on top of that, I have even more questions:
Does a smart pointer need to be generic in regard to the managed "target" (in case it's implementing Deref, that means Deref::Target is just a generic type U, e.g. as passed as a type parameter to Arc<U>, Rc<U>, MutexGuard<U>, etc.)
Does a smart pointer always need to provide a non-failing dereferencing mechanism? (If not, then maybe Mutex<U> is also a smart pointer?)
Is it reasonable to implement Deref for non-smart-pointers? If yes, under which circumstances?
The reference mentions "smart pointers" in the "pointer types" subsection. Should the reference give any definition or is that part of the reference non-normative anyway? (In the second case, maybe a short note could be added to that paragraph?)
And most importantly… I still have no clear answer on the main question: "What are smart pointers?"
This is important as long as the Deref documentation warns us to not implement Deref for any non-smart-pointers. So maybe that documentation should be adjusted/improved?
I think the warning is more advice rather than an absolute rule. It's to "avoid confusion". You could probably cause confusion by overloading other operators in evil ways, evil that is to anyone trying to understand the program. Macros can probably also be used to make code incomprehensible. Do we have any examples of "questionable" use of Deref? I think the discussion is a bit academic, although I have not followed it all that closely, so apologies if I am missing the point.
.
.
. On the other hand, 
(not entirely serious here, but perhaps that clarifies what I mean).