More compact numbers in JSON?

Hey everyone,

So I noticed that when passing data back and forth, for my purposes, all floating point numbers that I send (and I send a lot) contain far more decimal places than I'll ever need. I also store this JSON, so storage is much bigger too.

I could cut the size tremendously if if say, the numbers, had half the decimal places, or something like that.

I tried using declaring my fields with the type half::f16 but it made no impact on the underlying JSON.

I think even serde_json::Value uses f64 for numbers?

So how do you think I could do this?

Or do you believe I shouldn't? (I don't need so much decimal place accuracy)

Here's a similar discussion with a solution provided by David Tolnay: Floats are serialized with unnecessary decimal place · Issue #562 · serde-rs/json · GitHub

1 Like

Interesting! Do you know of an example anywhere I can learn from? I wouldn't know where to begin with this with_formatter

Here's an example for a "date formatting" from the official docs:

https://serde.rs/custom-date-format.html

Note that this is an example for using Serde's serialize_with procedural macro:

#[serde(with = "my_date_format")]

1 Like

JSON is not a storage format, so you shouldn't be using it for storage in the first place. If you are trying to store serialized data, and space is a concern, then use a binary format that was specifically designed for compactness, eg. CBOR, Bincode, MsgPack, Protobuf, or BSON. (Better yet – shouldn't you be using a database?)

Text formats aren't going to be compact, and most serializers won't let you cut the number of decimal places etc. But if you still want to use a human-readable format for storage, you can just compress it using eg. gzip (deflate), that will typically lead to at least a 2-3 fold reduction in size.

2 Likes

Oh but this fundamentally changes the data type in the JSON to a string...
(notice the quotes for the field compact)

mod compact_float {
    //! rounds a float to 3 decimal places, when serialized into a str, such as for JSON
    //! offsers space savings when such such precision is not needed.
    use serde::{Deserialize, Deserializer, Serializer};

    pub fn serialize<S>(float: &f32, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let s = format!("{:.3}", float);
        serializer.serialize_str(&s)
    }

    pub fn deserialize<'de, D>(deserializer: D) -> Result<f32, D::Error>
    where
        D: Deserializer<'de>,
    {
        let s = String::deserialize(deserializer)?;
        s.parse::<f32>().map_err(serde::de::Error::custom)
    }
}

#[cfg(test)]
mod tests {
    use super::compact_float;
    use serde::{Deserialize, Serialize};

    #[derive(Deserialize, Serialize, Debug)]
    struct StructWithCompactFloat {
        #[serde(with = "compact_float")]
        compact: f32,
        regular: f32,
    }

    #[test]
    fn floats_are_f32_internally_but_sent_out_as_compact() {
        let json_str = r#"
            {
                "compact": "0.234567",
                "regular": 0.234567
            }
        "#;

        let data: StructWithCompactFloat = serde_json::from_str(json_str).unwrap();
        println!("{:#?}", data);

        let serialized = serde_json::to_string_pretty(&data).unwrap();
        println!("{}", serialized);
    }
}

...I guess, I could use this trick to multiply by u16::MAX or so (since the numbers will always be in the range of 0.0..1.0), then serialize to u16
... and then do the opposite when deserializing

That's because you are serializing it like a string in:

serializer.serialize_str(&s)

The Serializer trait has other methods such as serialize_f32 that is what you are looking for:

mod compact_float {
    //! rounds a float to 3 decimal places, when serialized into a str, such as for JSON
    //! offsers space savings when such such precision is not needed.
    use serde::{Deserialize, Deserializer, Serializer};

    pub fn serialize<S>(float: &f32, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let s = format!("{:.3}", float);
        let parsed = s.parse::<f32>().unwrap();
        serializer.serialize_f32(parsed)
    }

    pub fn deserialize<'de, D>(deserializer: D) -> Result<f32, D::Error>
    where
        D: Deserializer<'de>,
    {
        f32::deserialize(deserializer)
    }
}
2 Likes