Confused by use of self, in example in chapter 17 of Rust book (2nd edition)

help
1

I'm reading chapter 17 of the Rust programming language, and I'm confused by the example in the 3rd section ("Object-Oriented Design Pattern Implementation").
Specifically listing 17-15, here.

There several confusing things with the code sample in this chapter, but the 2 main ones are:

  1. Why is Option used in this Post struct?

    pub struct Post {
    state: Option<Box>,
    content: String,
    }

    impl Post {
    pub fn request_review(&mut self) {
    if let Some(s) = self.state.take() {
    self.state = Some(s.request_review())
    }
    }
    }

  2. What is this syntax self: Box<Self> used here, and is it related to the use of Option? The text seems to hint that this is the case, but doesn't directly explain it at all.

trait State {
    fn request_review(self: Box<Self>) -> Box<State>;
}

This syntax self: Box<Self> for a method is confusing, since it's alluded in chapter 5 section 3 that methods are defined with self, &self or &mut self as parameters. Seeing suddenly self: Box<Self> as another way to define a method, made me think that methods could be defined in other ways, but trying to write self : &Box<Self> failed to compile. This also doesn't match my intuition from C++. It looks like a form of pattern matching on self, but apparently isn't (or else &Box<Self> would also work).

The only place I found where self : Box<Self> as a way to define a method was an an old stackoverflow question from 2014, but the book doesn't mention it nor any of the references.

So why did I try to write self : &Box<Self> ? Because it seems to me that if this were possible, it would make the Option in Post unnecessary, as well as the unwrapping / wrapping going on in the request_review method in the Post impl.

The way I understand this is something like this:

  1. We want to use a trait object to get a polymorphic behavior at runtime.

  2. To do this, we define the State as a trait, and put it inside a Box (in struct Post) to make it a trait object.

  3. We also define a method on the trait taking this trait object (request_review in the trait).

  4. We couldn't use &mut self since then (I think) we wouldn't be able to return a polymorphic result in a Box - I checked and saw that in the trait implementation, if I used a &mut self, I couldn't return another implementation, that is, this won't compile (I'm still not sure why exactly, isn't &mut self implicitly of type Self, which is polymorphic?):

    trait State {
    fn test_test(&mut self);
    }
    struct Draft {}
    struct PendingReview {}

    impl State for Draft {
    fn test_test(&mut self) {
    self = &mut PendingReview{};
    }
    }

  5. So, the next best thing would be to use self : &mut Box<Self> since that would allow us to directly change what's inside the box without moving it. But that's not possible in a method, apparently.

  6. So, the next best thing is to use self : Box<Self>, which is a valid method syntax.

  7. However, this means the self is moved. We must move it somewhere so we wrap it up in an Option (possibly we could wrap it in any other struct? Why is Option important here? It doesn't seem to serve any purpose except as a place to move the boxed self to and from).

Am I right? Or did I miss something?

1 Like
2

I'm taking a stab at answering a couple of these questions without having read the chapter.

Presumably to represent a "nullable" value. In Java, for example, the Post.state variable may not always have a State value, and could be set to null. Since Rust has no notion of a null value, the Option enum is used instead.

Maybe this syntax has something to do with State being a trait instead of a struct, I'm not sure.

However, I'm fairly certain that the syntax...

impl Foo {
    fn foo(self) {}
    fn bar(&self) {}
    fn baz(&mut self) {}
}

... is syntactic sugar for the more verbose and legitimate syntax...

impl Foo {
    fn foo(self : Self) {}
    fn bar(self : &Self) {}
    fn baz(self : &mut Self) {}
}

playground example that compiles: https://is.gd/vdZEDt

I'm not sure why you may have ran into a compiler error.

2 Likes
3

Using &mut self would imply the more verbose syntax of...

trait State {
    fn test_state(self : &mut Self) {}
}

...but what is Self in this context? I think it refers to the implementing struct of trait State, whatever that may be.

In your example...

impl State for Draft {
	fn test_test(&mut self) {
		self = &mut PendingReview{};
	}
}

test_test has one parameter self : &mut Self, which is self : &mut Draft, which creates a conflict of types when you try to assign PendingReview to self.

I think this is why the example in Chapter 17[1] chose to consume the old state and return a new state.

4

Yes, a value is moved from the Option into self when request_review is called. Right now Post owns the Option, which in turn owns the State trait object.

Rust won't like you taking the State value away, because what's left in its place in memory? The Chapter relies on Option.take to transfer ownership of the contained value and convert the Option owned by Post into a None so that Post never holds an invalid value.

I hope this helps. I thought smaller posts would be easier to read, but Discourse just educated me otherwise. Sorry for the noise.

1 Like
5

The books explains it:

This is where the Option comes in: we're going to take the Some value out of the state field and leave a None in its place since Rust doesn't let us have unpopulated fields in structs. Then we'll set the post's state value to the result of this operation.

The thing is, you "cannot move out of borrowed content". If you have an object, you can't move a field out of it and leave nothing in its place -- even if you're planning to put something back later. Every object must always contain a valid value.

What I've just said is not entirely true. As a special case, if you're working with an object that's on the stack, that is, if you own it, then Rust will allow you to move separate fields out, unless the type implements Drop.

But if you don't own the value, and request_review() only takes &mut self, you cannot do that. So instead they wrap the value in an Option and use its .take() method to move the value, leaving None instead. (I want to emphasize that leaving nothing, i.e. not a valid value is a very different thing to leaving None, a valid value of type Option<T>).

Then they process the value, wrap the result in Some() and put it back.

It seems you are familiar with C++, so let me give a C++ example. If you were to write this in C++, it would be something like:

class State {
public:
    // notice the move semantics!
    State request_review() && {
        ...
    }
};

class Post {
    std::unique_ptr<State> state;
    std::string content;
public:
    void request_review() {
        // the following line "moves" state
        // in C++, unlike in Rust, move is not a real move
        // instead, after the "move", state will be nullptr,
        // but stat's still a valid placeholder value you can access,
        // not "no value at all" like it is in Rust
        //
        // you can think of it as:
        // s = state;
        // state = nullptr;
        // state = s->request_review();
        state = state->request_review();
        // in Rust, the pointer nullability is explicit
        // with Option wrapping some reference type
    }
};

No, it's not related to the Option. In fact, the State has no idea Post wraps it in an Option. Actually, the book explains Box<Self> too:

Note that rather than having self, &self, or &mut self as the first parameter of the method, we have self: Box<Self>. This syntax means the method is only valid when called on a Box holding the type. This syntax takes ownership of Box<Self>, which is what we want because we're transforming the old state into a new state, and we want the old state to no longer be valid.

The implementation for the request_review method on Draft is to return a new, boxed instance of the PendingReview struct, which is a new type we've introduced that represents the state when a post is waiting for a review. The PendingReview struct also implements the request_review method <...>

(to be continued)

3 Likes
6

I believe right now there are four ways to declare a method with self:

fn foo(self: Self)       // shorthand: self
fn foo(self: &Self)      // shorthand: &self
fn foo(self: &mut Self)  // shorthand: &mut self
fn foo(self: Box<Self>)

Note that you don't have to use the Self keyword, e.g. in impl Foo you can write fn bar(self: &mut Foo) explicitly, but in a trait Self is the only way you refer to the type implementing it. You have to use self, however, otherwise you won't be able to call methods like.so(), only Foo::so(like) -- and the &self shorthands are only available on self, not other arguments.

Now, it would certainly be useful to have the ability to use other types with self, for example self: Rc<Self> and self: *mut Self. So, 2 days ago the language team discussed this feature and decided to work on it, "but it's not the highest priority at the moment."

That all being said, &Box<Self> isn't really useful -- use &self instead.

Because this would break:

trait Foo {}
struct Bar {}
struct Baz {}
impl Foo for Bar {}
impl Foo for Baz {}

fn takes_mut_foo(foo: &mut Foo) {
    *foo = Baz;
}

fn main() {
    let mut bar = Bar;
    takes_mut_foo(&mut bar);
    // WTF, is bar of type Baz now
    // or just an incorrect value of type Bar?
}

Generally, you cannot access the "value" of a trait object (or any dynamically sized type), only call methods on it. This includes overwriting the value.

Yes, &mut Box<Self> would be useful here, except it's not entirely clear how the implementation of Self would be determined ("Have to work out how to find the vtable for trait objects" point from the link above).

Yeah, we must move it somewhere, but that's not a problem. The problem we're solving with Option is that we must leave something in the place we're moving from.

Option is just the most convenient and idiomatic way to represent either a value or its absence (while it's moved out). You could use Result<Box<State>, ()> (which is, in a way, the same), or, if Box was nullable, its null value (but it isn't, and we use Option<Box<T>> instead, that's the whole point).

2 Likes
7

Thank you for this explanation! it made several things clearer for me. I went back and experimented some more based on your answers and I think I understand all this better now.

I found I made an error, which probably confused you: when I wrote that the code didn't compile, I meant that the code accepting a self : &Box<Self> didn't compile, but a paragraph above that I wrote that I found out it's apparently a legit syntax based on some SO link. But that link doesn't say that, I meant to say that (a) self : &Box<Self> doesn't compile and (b) self : Box<Self> does compile but it was a non-intuitive way (IMO) to define a method, compared to the 3 methods you listed (in your foo example).

I'll check if I can still edit my post and correct this.

I also didn't understand why a parameter of type &mut self isn't polymorphic, but a parameter of type self : Box<Self> is polymorphic, but I think I understand this better now thanks to your explanation and bugaevc's - Self by itself is never polymorphic, only when it's inside a Box (or Arc, etc).

8

Thanks for your reply, it made many things click for me!
Especially for the C++ snippet which really helped.

One thing, though:

Hmm, unless I'm misunderstanding something, self: Box<Self> creates the "problem" of request_review (the State trait method) taking ownership of its self param, which creates a problem at the call site in request_review (the Post method), since it's moving away a part of the Post struct which isn't owned, which, in turn, requires the creative solution of modifying the Post struct itself to have the Box wrapped up somehow, and Option is the most convenient and idiomatic solution for wrapping it (although there are others, as you said).

So unless I'm missing something, it's all connected.

I'll try to answer my own question then:
The use of Option here is an idiomatic trick to overcome a minor issue with the Rust language, where the only way to call a method on a trait object polymorphically (self : Box<Self>), also forces the called method to take ownership of the object. If this object is a field within another object the calling function doesn't own, this creates an ownership problem, since the calling function would then take a field outside a struct. We can't do that.

So wrapping the trait object with an Option, and using its take() method, allows us to sidestep this limitation by (a) moving the trait object to a local variable in the calling function (so now it owns it - done by the let if Some(s) binding), (b) call the trait object's method: s.request_review() (which then takes ownership of s, but that's ok since it's now local and not inside a struct we don't own) (c) the called method (in this case) returns a new Boxed object, which the calling method immediately wraps in Some, so at this point we own Some and Some owns the new object, and then (e) we modify the state field of the Post struct and transfer ownership to it.

I refer to it as a trick, since as far as I can see, Option is used just to appease the borrow checker: it isn't used in its "normative" sense to indicate the presence of absence of value, since the word "indicate" implies (to me) that there's some other piece of code in the program which cares about it. In other words, to observe a presence or an absence one needs an observer, but there's no observer here. The borrow checker doesn't care if there's a value there or not, it just needs to see that proper ownership rules are followed, and we use the take() method for that, since it's a succinct way of doing that.

Well, maybe a bit too succinct... this example and the some parts of the text trying to explain it in the book felt slightly misleading (well, to me at least). But thanks to boxofrox and bugaevc above I think I understand it better now. Thanks!

9

Yes, it is all connected. What I meant is we use Box<Self> because of other reasons (possibly replacing a value of one type with a value of another), not because of that Option. Then, in turn, we use the Option because of the Box (namely, because it needs to be moved, which would not be the case with a reference). Not the other way around.

No-no-no. You can totally call &self and &mut self methods on trait objects (getting all the dynamic dispatch and polymorphism), and in fact 99% of the time that's what you need, self: Box<Self> is really a very rarely needed language feature, with trait objects or not.

The reason we have to use Box<Self> here is because we're replacing the object with another object of a different type. Let me give a C++ example once again:

class Trait {
public:
    virtual void method() = 0;
};

class Foo : public Trait {
public:
    void method() override {
        // code 1
    }
};

class Bar : public Trait {
public:
    void method() override {
        // code 2
    }
};

// it should be Trait::method() that does this, but there's no Box<Self> analogue in C++

// this *won't* work
void inner1(Trait &t) {
    t = Bar();
}
void outer1() {
    Foo foo = Foo();
    inner1(foo);
    // whoops
}

// this will work
std::unique_ptr<Trait> inner2(std::unique_ptr<Trait> t) {
    return std::make_unique<Bar>();
}
void outer2() {
    std::unique_prt<Trait> first_foo_then_bar = std::make_unique<Foo>();
    first_foo_then_bar = inner2(first_foo_then_bar);
}

Well, the rule isn't in its place for no reason. Imagine the implementation of request_review() panics -- while the field is moved out -- and we start unwinding the stack, dropping/deallocating all the objects. In this case, it is crucial not to try to run drop() on a field that's been moved out.

If your value lives on the stack, Rust tracks what fields you have moved out and only drops those fields that you haven't. But if you're borrowing someone else's value, they (i.e. the other piece of code) are responsible for cleaning it up, and they have no idea what parts of it your function has moved out. This is why you're not allowed to move out of borrowed content, even if you're planning to move something back in later.

If you workaround this, in this case, with an Option, drop-on-panic-wise everything is perfectly good and safe: the field will be dropped, but since it's just None, it won't go further and try to drop garbage that the moved out value would leave in its place.

1 Like
10

@bugaevc is doing a great job providing additional explanation here. Once everything is cleared up, I'd love issues on the book repo with suggestions for how to make this chapter clearer!

3 Likes
11

Thanks, I was under the mistaken impression self : Box<Self> is required for dynamic dispatch, that cleared it up. I'll fix my post above.

Also thank you for the C++ example.

But both your C++ code and what you said got me thinking: maybe it's possible to implement the same functionality using only &self and Box<State> (as opposed to self : Box<Self> and Option<Box<State>>) ?

I think I managed to do that pretty trivially.
Here's the full code, also on the Rust playground here

pub struct Post {
    //state : Option<Box<State>>, // unnecessary? :-)
    state : Box<State>,
    content : String,
}

impl Post {
    pub fn request_review(&mut self) {
        self.state = (*self.state).request_review();
    }
}

trait State {
    fn request_review(&self) -> Box<State>;
    fn print(&self);
}

struct Draft {}
struct PendingReview {} 

impl State for Draft {
    fn request_review(&self) -> Box<State> {
        Box::new(PendingReview{})
    }
    fn print(&self) {
        println!("Draft");
    }
}

impl State for PendingReview {
    fn request_review(&self) -> Box<State> {
        Box::new(PendingReview{})
    }
    fn print(&self) {
        println!("PendingReview");
    }
}

fn main() {
    let mut p = Post { state : Box::new(Draft{}) , content : String::new() };
    p.state.print();
    p.request_review();
    p.state.print();
    p.request_review();
    p.state.print();
}

(I added some println!() instead of going for implementing the Debug trait in order to keep it simple but still to prove to myself that it does work as expected).

This doesn't use self : Box<Self> and Option<Box<State>> , and does the same thing.
For me, it's a less confusing example of what this chapter was trying to show, and it sorts of confirms my intuition that Option wasn't really necessary here.

Now, one might object to this code creating a new boxed state when it's unnecessary, in the request_review impl for PendingReview. The original code just returned self there, and my code returns a Box::new(PendingReview{}).

But I think objecting to that is kind of missing the point that this is supposed to be a chapter about an object oriented design pattern, not about optimizing the number of allocations. And if this were a real application, I would profile and check whether this unnecessary memory allocation is really what causes performance problem. I mean, this application (if this were a real one, that is) already uses memory allocation when moving from state to state, right? So does this small extra allocation when staying on the same state matter? Is making the code slightly less readable by introducing a layer of extra moves in and out of the Option worth it?

But! I guess the point of this chapter could have been to show:

  • The self : Box<Self> use in a method
  • The way to avoid unneeded memory allocations by returning self
  • The way to use these two together as a pattern for idiomatically achieving dynamic dispatch.

However, this wasn't presented as such. I would have expected a simpler example first (like my own, or something even simpler), and then the code with self : Box<Self> shown as an improvement (if at all), with a short discussion on how it's improving.

Or alternatively just mention that self : Box<Self> is another form of declaring a method, and that it can be used to avoid the unnecessary allocations, and leave it as an exercise to the reader. :slight_smile:

Thank you, that was a good explanation for why moving out of a borrowed value isn't allowed in the face of a panic.

12

Sure, as soon as I feel I understand this more completely, I'll open an issue.

I've already provided in my post above a rough sketch of how I would have expected the example and the chapter to look like in order to be simpler to understand, in my opinion.

But like I said there I'm not sure if the purpose of showing the object oriented design pattern shown in the chapter was also to show the use of self: Box<Self> and how idiomatic this use is(which would then make it all necessary) . Because in my opinion my simpler example is easier to understand, but that's my opinion, and I haven't written any Rust code or seen any "real" Rust code, I've literally just started to learn it... so I might be totally wrong.

13

That's actually is a great idea I haven't thought about!

...because it kind of misses one important point of the design presented in this chapter. State::request_review() is supposed to move the self, because it's meant to take it away from you, to statically enforce that you can only use this API in the correct way (replacing the old state with new). Now, it's debatable how applicable is this pattern to this problem, but generally using Rust's powerful type system to enforce correct API usage is a very idiomatic thing to do.

See the Mutex/MutexGuard API for a good example (the data a mutex protects is stored inside that mutex and you can only get access to it while you lock the mutex). Read this blogpost for another example.

By the way, Box<State> is a rather expensive thing to work with, but you can use this pattern with all sorts of objects, including zero-sized ones (I'm not sure the book describes those; this Rustonomicon page has some info). In that case, the object holds no meaningful value; it just plays the role of the ownership token. If you have it, you can use some API, otherwise you can't. And those APIs can enforce whether it's enough to have this token (if they take it by reference), whether you need to be its sole owner (by mutable reference), or that and in addition you can use it only once (move it).

Also, compare that to Mach port rights (receive, send, send-once) and how that enables building interesting APIs.

I believe adding #[derive(Debug)] is way shorter than manually implementing print(), though you'd also need to make State: Debug.

(Sorry for being brief, I'm kind of in a hurry)

14

@bugaevc beat me to it, but the semantics are different as he mentioned. With the Option case, you can only call request_review once per state - after first call, the option flips to None. If you forget to assign back a new value, that's the end of the line (unless something else can come along later and seed it again, but that's moving away from the spirit of the example).

In the Box<State> case, you can call request_review many times on the same state value, and if you don't assign it back, you're not transitioning anywhere. In fact, you're staying at some seemingly valid state that future caller may interpret correctly; a None would make the bug much more obvious.

15

Ok, but I'm slightly confused here. What exactly is the API you're referring to?

If the API is the functions marked as pub , then State::request_review is not in the API, rather it's an internal implementation detail of the state machine, so saying it's forcing me to take the self away for me is, I think, a bit useless, since I'm both the writer and the user of this code.

I think the only API here are the methods on Post, which are marked as pub. They don't take anything from the caller since they all accept &mut self.

Or perhaps you meant something else? Could you give me an example of a bug which can be caught in compile time, by using Option<Box<State>> rather than Box<State>?

Also, a Box can't be emptied, right? So it's impossible to leave the state as empty anyway if we only use Box without an Option. So it might be argued that wrapping it in an Option only makes the code more error prone, not less...

But maybe you were referring to the second section of 17.3?

I mean, this section of the book (17.3) has 2 separate sub-sections, with 2 implementations of the same state machine: one using dynamic dispatch, and one without dynamic dispatch, using only the type system to enforce variants (e.g. see listing 17-20).
In my original question and in all subsequent discussion, I was only confused about the first implementation, the second one is very clear.

And this second implementation, using the type system, does look very much like it embodies this idea you're talking about, that is, its pub methods take away self from the caller and return a new state.

Also compare listing 17-11 (which exercises the API for the first version) with listing 17-21 (which exercises the API for the second version).
Listing 17-11 does this to advance the state:

let mut post = Post::new();
post.add_text("I ate a salad for lunch today");
assert_eq!("", post.content());
post.request_review();
assert_eq!("", post.content());

Listing 17-21 does this to advance the state:

let mut post = Post::new();
post.add_text("I ate a salad for lunch today");
let post = post.request_review();
let post = post.approve();

This looks more like the thing you were talking about - forcing the API user's to call it the right way. Which the first example doesn't.

That blog post was a great read, thanks!
But unless I've misunderstood it, it's similar to the second example in chapter 17, in that it also advocates using the type system to enforce the state machine invariants in compile time. So does the Mutex/Mutex Guard API.

And by the way, I'm all for it! I'd much rather have compile time checks than dynamic behavior at runtime exploding... But the first example in that section is all about the dynamic behavior, and I still think that using Option there in what amounts to an internal implementation detail brings little benefit.

And also by the way, if @carols10cents is reading, I think that this section does a great job in comparing and contrasting the two implementations (dynamic vs. static) and it presents a honest comparison of both.

I tried doing just that (making State implement Debug) but couldn't figure out exactly how to.

Thanks for your time! :slight_smile:

1 Like
16

I'm sorry, but I don't quite understand what you mean.
There are 2 possible functions called request_review in the first section of chapter 17.3, I'm not sure which one you're referring to.
If you're referring to the "API" version (the one on Post marked as pub), then it's entirely possible to call it twice on the same state. It also doesn't flips the option to None and leaves it this way, as far as I can see, it only does this "behind the back" of the caller, regardless of whether it's a new state or an old state.

Here's an example where I take the original code and I call request_review twice on the same state (see the main() function).

On the other hand, if you're referring to the "implementation" version of request_review, then it doesn't flip the option to None either (again, except very temporarily, in a way which I don't think prevents anything).

So... I think you mean that there's some way in which it's possible to take the code of listing 17-15, write a user code against the public API (i.e., the request_review method on the Post struct), and get a None.
But it must be something in Rust I don't fully understand yet, because I see no such way. Could you show me how this could be possible? Thanks :slight_smile:

17

I was referring to Post::request_review. My point was if you made the mistake inside that method of not assigning a new state (the book doesn't make this error, but that wasn't what I was trying to illustrate), the internal Option would stay as None. That would be better detected later since that's an invalid state.

With a plain Box, if you don't assign back you're left with a seemingly valid state but the state machine is broken in a subtle way.

I should also clarify that the book example doesn't detect the None because it does nothing in that case. My point was more general, and not strictly using the book example.

18

Yes, I was referring to the State API. Again, this may not be the best example, 'cause indeed State is not a public API but an internal implementation detail, and the whole example is pretty small (but then again it's an example, of course they're not going to show you a full-featured codebase). The point is that State::request_review() taking the (boxed) self by value statically enforces that the caller (Post) does not use it anymore (after the review has been requested).

Yes, that is another example of how one can apply these ideas to API design (public API this time).

Exactly! Except, well, Mutex isn't quite a state machine. The point was that in Rust, the Mutex is explicitly tied to the data it protects, and the correctness is statically enforced by the type system.

It's not quite making State implement Debug (a trait cannot implement another trait...)

use std::fmt::Debug;

// "State: Debug" means that any type that wants
// to implement State also needs to implement
// Debug. This is kind of like abstract class
// or interface inheritance.
trait State: Debug {
    fn request_review(&self) -> Box<State>;
}

// automagically implement Derive for these
// two structs

#[derive(Debug)]
struct Draft {}

#[derive(Debug)]
struct PendingReview {} 

// and then in main()
println!("{:?}", p.state);