Serde using 19.8KB (file size) to deserialise 29 f64s - are any optimisations possible

help
1

I have a struct which consists of 29 f64 fields, all but 2 of which are Optional. Serde works great, but it produces a lot of code, all up 19.5KB for this struct. Is that expected? It seems like a lot for what is a very simple and repetitive struct.

I made a manual deserialiser based on the example. And while it doesn't quite have all the same error protections (I'm not doing anything if the two non-optional fields are missing, leaving them as default values instead) it's a lot smaller, only 5.9KB.

My manual deserialiser 
// Based on https://serde.rs/deserialize-struct.html
impl<'de> Deserialize<'de> for Metrics {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where D: Deserializer<'de> {
        struct MetricsVistor;

        impl<'de> Visitor<'de> for MetricsVistor {
            type Value = Metrics;

            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                formatter.write_str("struct Duration")
            }

            fn visit_map<V>(self, mut map: V) -> Result<Metrics, V::Error>
            where
                V: MapAccess<'de>,
            {
                let mut new_metrics = Metrics::default();
                while let Some(key) = map.next_key::<&str>()? {
                    let value = map.next_value::<f64>()?;
                    match key {
                        "buffercharheight" => new_metrics.buffercharheight = Some(value),
                        "buffercharwidth" => new_metrics.buffercharwidth = Some(value),
                        "buffermargin" => new_metrics.buffermargin = Some(value),
                        "buffermarginx" => new_metrics.buffermarginx = Some(value),
                        "buffermarginy" => new_metrics.buffermarginy = Some(value),
                        "charheight" => new_metrics.charheight = Some(value),
                        "charwidth" => new_metrics.charheight = Some(value),
                        "graphicsmargin" => new_metrics.graphicsmargin = Some(value),
                        "graphicsmarginx" => new_metrics.graphicsmarginx = Some(value),
                        "graphicsmarginy" => new_metrics.graphicsmarginy = Some(value),
                        "gridcharheight" => new_metrics.gridcharheight = Some(value),
                        "gridcharwidth" => new_metrics.gridcharwidth = Some(value),
                        "gridmargin" => new_metrics.gridmargin = Some(value),
                        "gridmarginx" => new_metrics.gridmarginx = Some(value),
                        "gridmarginy" => new_metrics.gridmarginy = Some(value),
                        "height" => new_metrics.height = value,
                        "inspacing" => new_metrics.inspacing = Some(value),
                        "inspacingx" => new_metrics.inspacingx = Some(value),
                        "inspacingy" => new_metrics.inspacingy = Some(value),
                        "margin" => new_metrics.margin = Some(value),
                        "marginx" => new_metrics.marginx = Some(value),
                        "marginy" => new_metrics.marginy = Some(value),
                        "outspacing" => new_metrics.outspacing = Some(value),
                        "outspacingx" => new_metrics.outspacingx = Some(value),
                        "outspacingy" => new_metrics.outspacingy = Some(value),
                        "spacing" => new_metrics.spacing = Some(value),
                        "spacingx" => new_metrics.spacingx = Some(value),
                        "spacingy" => new_metrics.spacingy = Some(value),
                        "width" => new_metrics.width = value,
                        _ => {},
                    }
                }
                Ok(new_metrics)
            }
        }

        const FIELDS: &[&str] = &["buffercharheight", "buffercharwidth", "buffermargin", "buffermarginx", "buffermarginy", "charheight", "charwidth", "graphicsmargin", "graphicsmarginx", "graphicsmarginy", "gridcharheight", "gridcharwidth", "gridmargin", "gridmarginx", "gridmarginy", "height", "inspacing", "inspacingx", "inspacingy", "margin", "marginx", "marginy", "outspacing", "outspacingx", "outspacingy", "spacing", "spacingx", "spacingy", "width"];
        deserializer.deserialize_struct("Metrics", FIELDS, MetricsVistor)
    }
}

I'm guessing it can't automatically recognise and optimise that all the fields are the same type? Would it be reasonable to have better optimisations for structs like this in the future, or does this really necessitate a manual deserialiser?

20KB filesize is normally nothing to care much about, but I'm compiling to WASM and there are a lot of large Serde functions, this just was the most straightforward to try a manual approach with.

2 Likes
2

serde is designed for maximum flexibility and portability, thus the generated code is not particularly well optimized, for example, there are a lot of generics and recursions.

have you used miniserde? might worth a try.

https://crates.io/crates/miniserde

3

I heavily use enums with data in their variants, so I didn't think miniserde was an option.

Unless I could just use it for select structs like this one? Can serde and miniserde work together? Could I say that this Metrics struct is to be deserialised by miniserde even though its container struct would be deserialised by serde?

4

in thoery, it should be possible, but I don't know any existing solution to bridge these two, so you'll have to do a lot of manual work, which would probably be even more work than manually implementing Deserialize for your type, so I would not suggest this approach.

as for your manual implemented deserialization function, I think it's pretty much the best you can do. the only potential "improvement" I can think of would be to use a macro to slightly reduce the code duplication, but macros are generally harder to read, so it might as well end up to be no improvement at all.

btw, if enums is the only blocker for you to use miniserde, there's a crate miniserde-enum, which provides a derive macro for enum types that are not directly supported by miniserde, but I have not used it personally, so I don't know how well it works.

1 Like
5

I have to ask, sorry if it's obvious.

When checking the code size, did you build in release mode? That is, cargo build --release or rustc -C opt-level=3. The default dev/debug profile can generate very large code.

Link-time optimization (LTO) can also help in reducing code size.

4 Likes
6

Yep, release mode, the Rust static lib is compiled with LTO and code-units=1, and the final app is compiled with LLVM's LTO too. (Though I haven't got cross language LTO working.)

1 Like
7

Is this the size of generated code, or size of relevant section in the binary after compilation? Have you actually measured how much space in the final binary is occupied by this deserialization related code?

1 Like
8

That doesn't just miss error protections, but is also not capable for deserializing with some Deserializer's as it doesn't handle non-self describing serialization formats where generally visit_seq is called for structs rather than visit_map.

By the way which serialization crate are you combining your serde Deserialize implementation with? After optimizations de Deserializer and Deserialize impls get (partially) merged together.

9

True.. the example did cover that but as I only need to get it working with JSON I didn't implement the visit_seq method.

After optimizations de Deserializer and Deserialize impls get (partially) merged together.

Interesting, I haven't seen anyone mention that before. Of course it would make sense in terms of basic inlining.

It's the size of the functions in the binary. Using Twiggy I got these results:

15714 ┊     1.30% ┊ <remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::<serde::private::de::content::ContentDeserializer<serde_json::error::Error>>
 2804 ┊     0.23% ┊ <<remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::__FieldVisitor as serde_core::de::Visitor>::visit_bytes::<serde_json::error::Error>
  774 ┊     0.06% ┊ <<remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::__FieldVisitor as serde_core::de::Visitor>::visit_str::<serde_json::error::Error>
  208 ┊     0.02% ┊ <<remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Expected>::fmt

With my new manual deserialiser:

5329 ┊     0.50% ┊ <core::marker::PhantomData<remglk::glkapi::protocol::Metrics> as serde_core::de::DeserializeSeed>::deserialize::<serde::private::de::content::ContentDeserializer<serde_json::error::Error>>
 804 ┊     0.08% ┊ serde::private::de::content::visit_content_seq::<<remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::MetricsVistor, serde_json::error::Error>
 208 ┊     0.02% ┊ <<remglk::glkapi::protocol::Metrics as serde_core::de::Deserialize>::deserialize::MetricsVistor as serde_core::de::Expected>::fmt
10

Switching to a binary format rather than a self describing textual format like json would likely result in more compact code too. Something like postcard or bincode perhaps. Might not be an option, if you don't control both sides of the code.

1 Like
11

Yeah unfortunately in this case I need to consume and produce JSON.

12

Unfortunately proc macros don't see actual types when generating code, so they have limited ability to be smarter.