Some abstractions cannot be implemented

Yandros · February 22, 2022, 5:25pm

Making a trait `dyn` safe

trait Codec {
    fn encode<T: Serialize> (&self, arg: &'_ T)
      -> Result<Vec<u8>, Error>
    ;
    fn decode<T: DeserializeOwned> (&self, arg: &[u8])
      -> Result<T, Error>
    ;
}

1- A simple (but verbose) solution for simple cases

Usually, the downstream usage will not need to be as generic as the generic trait API is. Say you'll want to serialize a String and/or a i32.

In that case, you can very easily write a helper trait with those specific hard-coded choices of generics / specific monomorphisations:

trait DynCodec {
    fn encode_str (&self, arg: &'_ str)
      -> Result<Vec<u8>, Error>
    ;
    fn encode_i32 (&self, arg: i32)
      -> Result<Vec<u8>, Error>
    ;
    fn decode_string (&self, arg: &[u8])
      -> Result<String, Error>
    ;
    fn decode_i32 (&self, arg: &[u8])
      -> Result<i32, Error>
    ;
}
// From the general `Codec`, get this narrower `DynCodec`
impl<T : Codec> DynCodec for T {
    fn encode_str (&self, arg: &'_ str)
      -> Result<Vec<u8>, Error>
    {
        self.encode::<&str>(&arg) // extra indirection because of missing `?Sized` on the original `<T>` generic.
    }

    fn encode_i32 (&self, arg: i32)
      -> Result<Vec<u8>, Error>
    {
        self.encode::<i32>(arg)
    }

    fn decode_string (&self, arg: &[u8])
      -> Result<String, Error>
    {
        self.decode::<String>(arg)
    }

    fn decode_i32 (&self, arg: &[u8])
      -> Result<i32, Error>
    {
        self.decode::<String>(arg)
    }
}

and then use Box<dyn DynCodec + …> for Fun And Profit (any Box<impl 'lt + Codec> will magically coerce to Box<dyn 'lt + DynCodec>).

This does have the drawback of being cumbersome to write, but when that happens with Rust macros can be quite effective at palliating it.

2- Generalizing/extending this solution

So, the previous approach had two issues:

it only handled a fixed number of types,
it gets unwieldy as the number of types grows / does not scale to increasing that fixed number of types.

Both aspects have a nice solution: only use one type! Indeed, it's not because you only use one type that you have to give up on polymorphism: that's exactly what dyn Traits are for!

Hence:

/// Let's only handle `encode` for the moment
trait DynCodec {
    fn dyn_encode (&self, arg: &'_ dyn Serialize)
      -> Result<Vec<u8>, Error>
    ;
    /* decode not handled yet */
}

impl<T : Codec> DynCodec for Codec {
    fn dyn_encode (&self, arg: &'_ dyn Serialize)
      -> Result<Vec<u8>, Error>
    {
        self.encode(&arg)
    }
}

And then you can make a Box<dyn DynCodec + …>, and call .dyn_encode(&some_str), or .dyn_encode(&some_integer) on it, etc. Indeed, a &some_str can coerce that one &dyn Serialize type, and so can &some_integer.

And more generally, for any T : Serialize, a &T can coerce to a&dyn Serialize … which means we can go back to featuring an ergonomic generic façade atop our dyn_encode method!

impl Codec for dyn DynCodec + '_ {
    fn encode<T : Serialize> (&self, arg: &T)
      -> Result<Vec<u8>, Error>
    {
        self.dyn_encode(arg) // `arg as &dyn Serialize`
    }
}

There is only one caveat, though… ::serde::Serialize is not dyn-safe / can't be made into a dyn Trait.

Seeing this error can be disheartening, since we seem to be back to square one. But it's actually not the case: we needed a dyn Codec-like thing, and we got it, provided we had a dyn Serialize-like thing. And since we now know the recipe to make a Trait become dyn Trait-like thing, we can actually rinse and repeat with Serialize:

Find a restricted set of APIs to use in a non-generic fashion (if a dyn Trait can be found for that part, e.g., a dyn Serializer, then even better);
Write a DynSerialize trait with only those non-generic methods, and a blanket impl for all the impl Serialize types.
and so on…

Luckily, in our case, the ::serde framework itself has gotten us covered, since they do feature their companion crate:

which is basically all this approach already written in a maximally versatile and efficient manner, thanks to taking advantage of the foundations of the serde model being covered by a fixed number of root cases (serde bool, serde u8, serde sequence of serdables, serde map of serdables, etc.).

3- Fixing the `dyn Serialize` problem using `::erased-serde`

/// Let's only handle `encode` for the moment
trait DynCodec {
-   fn dyn_encode (&self, arg: &'_ dyn Serialize)
+   fn dyn_encode (&self, arg: &'_ dyn ::erased_serde::Serialize)
      -> Result<Vec<u8>, Error>
    ;
    /* decode not handled yet */
}

impl<T : Codec> DynCodec for Codec {
-   fn dyn_encode (&self, arg: &'_ dyn Serialize)
+   fn dyn_encode (&self, arg: &'_ dyn ::erased_serde::Serialize)
      -> Result<Vec<u8>, Error>
    {
        self.encode(&arg)
    }
}

and voilà.

4- Handling `.dyn_decode()` in a polymorphic fashion.

This is the main thorny / genuinely challenging one.

For instance, notice how there is no Deserialize trait in ::erased-serde.

Indeed, we'd like to end up with generics once all this dance has been done, but if were to go down that road, we'd kind of need to provide some dynamic representation of the type we'd like to decode (e.g., some form of TypeId parameter), to then get a Box<dyn Decoded> kind of dynamic type, to then go back to downcasting or something. It wouldn't be pretty, nor efficient, nor nice.

So, how does ::erased-serde handle that / circumvent that problem? Thanks to, again, the fixed number of root case: "serde bool, serde i32, serde sequence of serdables, etc."

With it, we can actually have a dyn ::erased_serde::Deserializer, that is, a type-unified entity which can be queried for these root elements, and which thus makes it a ::serde::Serializer itself, that is, something that can eat generics for breakfast.

So, our "internally dyn-compatible" layer will be based of Deserializers, and then we'll go back to generic <T : DeserializeOwned> atop it:

trait Codec : DecodeMethod {
    // fn encode…

    // no decode here (see below for the default impl dance).
    // Instead, implementors are expected to provide the deserializer directly
    fn deserializer<'buf> (
        &'_ self,
        buf: &'buf [u8],
    ) -> Box<dyn ::erased_serde::Deserializer<'buf>> // + 'buf
    ;
}
/// `.decode()` default-implemented here:
impl<C : Codec> DecodeMethod for C {
    fn decode<T : DeserializeOwned> (
        &self,
        buf: &'_ [u8],
    ) -> Result<T, Error>
    {
        ::erased_serde::deserialize(&mut self.deserializer(buf))
    }
}
// where:
trait DecodeMethod {
    fn decode<'buf, T : DeserializeOwned> ( // this could be `T : Deserialize<'buf>` btw
        &self,
        buf: &'buf [u8],
    ) -> Result<T, Error>
    ;
}

And thus you'd need to adjust a bit the impls so that they yield the whole Deserializer rather than perform the deserializations directly:

pub struct JsonCodec {}

impl Codec for JsonCodec {
    // fn encode…

    fn deserializer<'buf> (
        &'_ self,
        buf: &'buf [u8],
    ) -> Box<dyn ::erased_serde::Deserializer<'buf>> // + 'buf
    {
        // 1- Get a concrete deserializer (a json one, here)
        let json_deserializer = ::serde_json::Deserializer::from_slice(buf);
        // 2- Box-dyn it, as shown in erased-serde.
        Box::new(<dyn ::erased_serde::Deserializer<'buf>>::erase(
            json_deserializer
        ))
    }
}

You'd get decode for free (incidentally factoring out the deserializer-agnostic code you had (the match arm etc.).

5- Bonus: avoiding the `Box` on the `dyn Deserializer`.

By using a callback-based / CPS / scoped API rather than returning the serializer. See

for more details about this approach.

Using the sugar from that crate, in pseudo-code, it would be about changing deserializer to be:

  fn deserializer (…)
-   -> Box<dyn Deserializer…
+   -> &'local mut dyn Deserializer…

More precisely

trait Codec : DecodeMethod {
    fn encode (&self, arg: &impl Serialize)
      -> Result…
    ;

    fn with_deserializer<'buf> (
        self: &'_ Self,
        buf: &'buf [u8],
        yield_: &'_ mut dyn for<'local> FnMut(
  /* -> */ &'local mut Deserializer<'buf>
        ),
    )
    ;
}
/// Default-impl of `decode`
impl<C : Codec> DecodeMethod for C {
    fn decode<'buf, T : Deserialize<'buf>> (
        self: &'_ Self,
        buf: &'buf [u8],
    ) -> Result<T, Error>
    {
        let mut ret = None;
        self.with_deserializer(buf, &mut |deserializer| ret = Some({
            ::erased_serde::deserialize(deserializer)
        }));
        ret.unwrap()
    }
}

with a concrete impl then being:

impl Codec for JsonCodec {
    // fn encode…

    fn with_deserializer<'buf> (
        self: &'_ Self,
        buf: &'buf [u8],
        yield_: &'_ mut dyn for<'local> FnMut(
  /* -> */ &'local mut Deserializer<'buf>
        ),
    )
    {
        // 1- Get a concrete deserializer (a json one, here)
        let json_deserializer = ::serde_json::Deserializer::from_slice(buf)
        // 2- "return" it `&mut dyn`ed
        yield_(&mut <dyn ::erased_serde::Deserialize<'buf>>::erase(
            json_deserializer
        ))
    }
}

Topic		Replies	Views
[Solved] Implement trait for struct with associated type help	4	439	August 14, 2019
Generic type constraint on `impl<T: ?> SomeStruct {...}` help	10	565	August 20, 2020
Trait for something generic, but errors could be specific for trait implementation help	3	294	October 1, 2022
Working around E207 "unconstrained type parameter" help	3	538	May 31, 2020
Why is GenericB not implemented for Anon_b help	5	458	January 12, 2023

Some abstractions cannot be implemented

Making a trait dyn safe

1- A simple (but verbose) solution for simple cases

2- Generalizing/extending this solution

3- Fixing the dyn Serialize problem using ::erased-serde

4- Handling .dyn_decode() in a polymorphic fashion.

5- Bonus: avoiding the Box on the dyn Deserializer.

Related Topics

Making a trait `dyn` safe

3- Fixing the `dyn Serialize` problem using `::erased-serde`

4- Handling `.dyn_decode()` in a polymorphic fashion.

5- Bonus: avoiding the `Box` on the `dyn Deserializer`.