Custom Serde deserialization for varint-encoded fields

help
1

Hey,

I'm building a proxy that must deserialize and serialize external packets.
Previously, each packet had its own deserialization method which was insanely inefficient so I wanted to migrate to Serde + Bincode but I have an issue decoding some encoded values and especially varints.

I want to be able to do this :

#[derive(Packet, Deserialize)]
struct ExamplePacket {
    some_string: String,
    #[serde(deserialize_with = "deserialize_varint")]
    some_int: i32,
}

From what I understand, I need to create a custom deserializer to support this encoding but Serde expects the length to be known before deserialization, which is impossible in this case since the only length indication is a bit in each byte telling if the next byte is a part of the value.

I've tried :

  • Using deserialize_seq but it assumes the first byte is the length which in this case is wrong and we end up losing the first byte.
  • Using deserialize_tuple but it requires the length, which is unknown.
  • Using deserialize_bytes but I get an Unexpected EOF error, which might be due to length once again.

I've reached the point where I don't really know where to search anymore, so any help is gladly appreciated. Thanks!

2

I don't have an answer for you... but I believe the Postcard format supports varints? To quote a section of their doc:

Conceptually, all varint(N) types encode data in a similar way when considering a stream of bytes:

  • The most significant bit of each stream byte is used as a "continuation" flag.
    • If the flag is 1, then this byte is NOT the last byte that comprises this varint
    • If the flag is 0, then this byte IS the last byte that comprises this varint

Anyway, the postcard crate works with Serde, so perhaps that might be a starting point to look at?

And just to confirm, what is the output format you are targeting? You mentioned Bincode; is that correct?

3

So, I made a mistake assuming those crates were for general binary serialization. I ended up having a ton of other issues, like different endianness in the same packet and other annoying stuff.

I ended up doing something similar to my old code but with macros and traits to generate the methods so it looks like the snippet I sent earlier, but without the external crates.