Help understanding extension trait for &Self and &mut Self references and Sized

[lancegatlin]

Hi folks!

I'm trying to understand why when using an extension trait to add methods, the "&mut Self" type gains +Sized requirement. My understanding of &mut T and &T is that since they are references whether T is Sized shouldn't matter since the size of the pointer is known at compile time.

trait ParserExt<E> : Parser<E> {
	fn parse_field_FAILS<I,O:ParseFrom<I,E>>(&mut self, field: String, from: I) -> Result<O,E> {
		self.begin_field();
		let retv = O::parse_from(self, from);
		self.end_field();
		retv
	}
}

Error:

error[E0277]: the size for values of type `Self` cannot be known at compilation time
  --> src/main.rs:13:34
   |
13 |         let retv = O::parse_from(self, from);
   |                                  ^^^^ doesn't have a size known at compile-time
   |
   = help: the trait `std::marker::Sized` is not implemented for `Self`
   = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
   = help: consider adding a `where Self: std::marker::Sized` bound
   = note: required for the cast to the object type `dyn Parser<E>`

If I make this method a normal function, it works fine:

fn parse_field_WORKS<E,I,O:ParseFrom<I,E>>(parser: &mut Parser<E>, field: String, from: I) -> Result<O,E> {
	parser.begin_field();
	let retv = O::parse_from(parser, from);
	parser.end_field();
	retv
}

Playground:

Rust Playground

Thanks!

[Yandros]

Given this trait definition:

trait ParseFrom<I, E> : Sized {
    fn parse_from (parser: &mut dyn Parser<E>, from: I) -> Result<Self, E>;
}

the first argument must be a mut reference to the dyn Parser<E> trait object, i.e., a &mut dyn Parser<E>, or something that can coerce to it: a mutable reference to a concrete type that implements Parser<E> (written &mut (impl Parser<E> + Sized)).

There is, however, a special and quite counter intuitive example here, and it deals with a "subtrait object", given a potential subtrait of Parser<E>:

trait SubParser<E> : Parser<E> {}

fn test<I, E, O : ParseFrom<I, E>> (input: &mut dyn SubParser<E>, from : I)
{
    let _ = O::parse_from(input, from); // Error, expected trait `Parser`, found trait `SubParser`
}

Now, at line 13 (linted by the compiler), you are using a self: &mut Self. This is saying that it needs to be possible to coerce &mut Self to a &mut dyn Parser. But when Self is allowed to be ?Sized, then this includes Self = dyn SubParser<E>, which, as shown above, does not work.

---

The solution? Instead of taking trait objects directly, take generics with the ?Sized bound:

#![deny(bare_trait_objects)]

trait ParseFrom<I, E> : Sized {
    fn parse_from (
        parser: &mut (impl Parser<E> + ?Sized),
        from: I,
    ) -> Result<Self, E>;
}

• Playground

[lancegatlin]

Thanks Yandros. I concur on solution, but in the absence of type alias for traits, this is very ugly and very viral. After applying ?Sized to a type, it leaks to all functions that use that type. ParseFrom trait impls is majority of code I'm writing now. I'd prefer a cleaner solution, which is why I was trying to use just &mut Parser< E> in ParseFrom.

In other languages, type extensions are simple sugar and I would expect parse_field_FAILS and parse_field_WORKS to be semantically equivalent, but that doesn't seem to be the case with Rust. I'm much more interested in understanding why using the extension trait fails here, as it shows me I'm either missing something fundamental about how Rust works or Rust has a shortcoming that I need to design around.

My apologies if you are not interested in discussing this to this level of detail. If so, I'm very happy to accept that as your answer.

@Yandros

Given this trait definition:

trait ParseFrom<I, E> : Sized {
    fn parse_from (parser: &mut dyn Parser<E>, from: I) -> Result<Self, E>;
}

the first argument must be a mut reference to the dyn Parser<E> trait object, i.e. , a &mut dyn Parser<E> , or something that can coerce to it: a mutable reference to a concrete type that implements Parser<E> (written &mut (impl Parser<E> + Sized) ).

Please check my understanding:

1. &mut Parser< E> is equivalent to &mut dyn Parser< E> and is how we use dynamic dispatch on a trait object. Here &mut Parser< E> means that I expect to receive (at runtime) a dynamic instance of Parser< E> and use dynamic dispatch to call it's methods.
2. (parser: &mut impl Parser< E>, ... ) is sugar for <P:Parser< E>>(parser: &mut P, ...), which is static dispatch of calls to methods of parser.
3. Given #1 and #2 above, I don't see how your statement above is true.

[lancegatlin]

@Yandros

There is, however, a special and quite counter intuitive example here, and it deals with a “subtrait object”, given a potential subtrait of Parser<E> :

trait SubParser<E> : Parser<E> {}

fn test<I, E, O : ParseFrom<I, E>> (input: &mut dyn SubParser<E>, from : I)
{
    let _ = O::parse_from(input, from); // Error, expected trait `Parser`, found trait `SubParser`
}

Now, at line 13 (linted by the compiler), you are using a self: &mut Self . This is saying that it needs to be possible to coerce &mut Self to a &mut dyn Parser . But when Self is allowed to be ?Sized , then this includes Self = dyn SubParser<E> , which, as shown above, does not work.

Ah now this is getting to the heart of the issue. Please check my understanding:

1. Traits do not inherit instead they act only as type bounds on the trait being declared. e.g trait ParserExt< E> : Parser< E> means only that to create an impl of ParserExt< E> for some type T then impl Parser< E> for T must already exist.
2. &mut Self in this case is of type ParserExt< E> and since there is no inheritance the compiler won't upcast this for me.

In your example above, is there a way to cast "input" to Parser< E> ?

[lancegatlin]

Some Google-Fu turned up some new ideas based on above. Here is the solution I'm going with that works around the issue without spreading ?Sized everywhere:

trait ParseFrom<I,E> : Sized {
	fn parse_from(parser: &mut Parser<E>, from: I) -> Result<Self,E>;
}

trait Parser<E> {
	fn as_parser_mut(&mut self) -> &mut Parser<E>;
	...
}

trait ParserExt<E> : Parser<E> {
	fn parse_field<I,O:ParseFrom<I,E>>(&mut self, field: String, from: I) -> Result<O,E> {
		self.begin_field();
		let retv = O::parse_from(self.as_parser_mut(), from);
		self.end_field();
		retv
	}
}
impl<E,P:Parser<E>> ParserExt<E> for P where P:?Sized { }

impl Parser<String> for DummyParser {
	fn as_parser_mut(&mut self) -> &mut Parser<String> { self }
	...
}

Playground

[Hyeonu]

As a side note, please not use &[mut] Trait syntax. It's deprecated and replaced by &[mut] dyn Trait. In Rust every method calls will be translated as direct function call which allows aggressive compiler optimization like inlining, to produce much faster code compared to what you actually wrote. The only exception is the "Trait object"(dyn Trait in new syntax) which builds "vtable"(struct of function pointers) on creation, and perform indirect call on invoke. This distinction is marginal for large functions, but can results N times speedup/down for small functions. To apply it in your code, add #![warn(bare_trait_object)] at the top of your lib/main.rs and run cargo check.

[lancegatlin]

Good to know Hyeonu. I've seen the dyn keyword, but the compiler (1.35) doesn't require it (or warn if it is left out). Also I don't remember seeing that in the Rust book or docs, thanks for the explanation.

[Yandros]

@lancegatlin

1. &mut Self in this case is of type ParserExt< E> and since there is no inheritance the compiler won’t upcast this for me.

In your example above, is there a way to cast “input” to Parser< E> ?

The main problem comes from upcasting the potentially existent dyn SubParser into a dyn Parser. Although this should be possible, it is currently not the case in Rust, since there are some unresolved issues about the precise mechanism to achieve this, see this thread and this other one.

@lancegatlin

Some Google-Fu turned up some new ideas based on above. Here is the solution I’m going with that works around the issue without spreading ?Sized everywhere:

#![deny(bare_trait_objects)]

trait ParseFrom<I,E> : Sized {
	fn parse_from (parser: &mut dyn Parser<E>, from: I) -> Result<Self,E>;
}

trait Parser<E> {
	fn as_parser_mut (&mut self) -> &mut dyn Parser<E>;
	...
}

This does indeed resolve the upcasting issue, since it adds an explicit method on the vtable in charge of performing the upcasting. It is annoying because it forces you to write the trivial implementation every time you want to implement the trait.

@lancegatlin

• &mut Parser< E> is equivalent to &mut dyn Parser< E> and is how we use dynamic dispatch on a trait object. Here &mut Parser< E> means that I expect to receive (at runtime) a dynamic instance of Parser< E> and use dynamic dispatch to call it’s methods.
• (parser: &mut impl Parser< E>, … ) is sugar for <P:Parser< E>>(parser: &mut P, …), which is static dispatch of calls to methods of parser.
• Given #1 and #2 above, I don’t see how your [following] statement is true.

@Yandros

the first argument must be a mut reference to the dyn Parser<E> trait object, i.e. , a &mut dyn Parser<E> , or something that can coerce to it: a mutable reference to a concrete type that implements Parser<E> (written &mut (impl Parser<E> + Sized) ).

My poor phrasing is to blame, here; the argument is required to be a &mut dyn Parser, which does indeed use dynamic dispatch based on a very precise vtable: dyn Parser's vtable (i.e., dyn SubParser's vtable does not do the trick here).

Now, dyn Parser implements Parser and so does dyn SubParser too. Meaning that both these types could be Self in your example, provided there is not a Self : Sized bound.

• My &mut impl Parser + ?Sized (i.e., <P : Parser + ?Sized> ... &mut P) solution on the called function allowed to solve the issue since then such function would be able to momonorphise (statically dispatch) to both the P = dyn Parser and P = dyn SubParser cases.

• And yes, it is true that having to add ?Sized for each generic is annoying it's a retrocompatibility artifact from Sized's current design not having existed at the beginning of Rust...

And when Self = dyn SubParser, &mut Self cannot be coerced to a &mut dyn Parser, since that would imply upcasting, which is not a conversion available in Rust directly; hence the requirement for a method such as as_parser_mut.

---

I hope it's clearer, now

[lancegatlin]

Yes thank you so much explaining all that!

Playground with dyn