More compact numbers in JSON?

[arifd]

Hey everyone,

So I noticed that when passing data back and forth, for my purposes, all floating point numbers that I send (and I send a lot) contain far more decimal places than I'll ever need. I also store this JSON, so storage is much bigger too.

I could cut the size tremendously if if say, the numbers, had half the decimal places, or something like that.

I tried using declaring my fields with the type half::f16 but it made no impact on the underlying JSON.

I think even serde_json::Value uses f64 for numbers?

So how do you think I could do this?

Or do you believe I shouldn't? (I don't need so much decimal place accuracy)

[moy2010]

Here's a similar discussion with a solution provided by David Tolnay: Floats are serialized with unnecessary decimal place · Issue #562 · serde-rs/json · GitHub

[arifd]

Interesting! Do you know of an example anywhere I can learn from? I wouldn't know where to begin with this with_formatter

[moy2010]

Here's an example for a "date formatting" from the official docs:

https://serde.rs/custom-date-format.html

Note that this is an example for using Serde's serialize_with procedural macro:

#[serde(with = "my_date_format")]

[H2CO3]

@arifd

I could cut the size tremendously if if say, the numbers, had half the decimal places, or something like that.

JSON is not a storage format, so you shouldn't be using it for storage in the first place. If you are trying to store serialized data, and space is a concern, then use a binary format that was specifically designed for compactness, eg. CBOR, Bincode, MsgPack, Protobuf, or BSON. (Better yet – shouldn't you be using a database?)

Text formats aren't going to be compact, and most serializers won't let you cut the number of decimal places etc. But if you still want to use a human-readable format for storage, you can just compress it using eg. gzip (deflate), that will typically lead to at least a 2-3 fold reduction in size.

[arifd]

Oh but this fundamentally changes the data type in the JSON to a string...
(notice the quotes for the field compact)

mod compact_float {
    //! rounds a float to 3 decimal places, when serialized into a str, such as for JSON
    //! offsers space savings when such such precision is not needed.
    use serde::{Deserialize, Deserializer, Serializer};

    pub fn serialize<S>(float: &f32, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let s = format!("{:.3}", float);
        serializer.serialize_str(&s)
    }

    pub fn deserialize<'de, D>(deserializer: D) -> Result<f32, D::Error>
    where
        D: Deserializer<'de>,
    {
        let s = String::deserialize(deserializer)?;
        s.parse::<f32>().map_err(serde::de::Error::custom)
    }
}

#[cfg(test)]
mod tests {
    use super::compact_float;
    use serde::{Deserialize, Serialize};

    #[derive(Deserialize, Serialize, Debug)]
    struct StructWithCompactFloat {
        #[serde(with = "compact_float")]
        compact: f32,
        regular: f32,
    }

    #[test]
    fn floats_are_f32_internally_but_sent_out_as_compact() {
        let json_str = r#"
            {
                "compact": "0.234567",
                "regular": 0.234567
            }
        "#;

        let data: StructWithCompactFloat = serde_json::from_str(json_str).unwrap();
        println!("{:#?}", data);

        let serialized = serde_json::to_string_pretty(&data).unwrap();
        println!("{}", serialized);
    }
}

...I guess, I could use this trick to multiply by u16::MAX or so (since the numbers will always be in the range of 0.0..1.0), then serialize to u16
... and then do the opposite when deserializing

[moy2010]

That's because you are serializing it like a string in:

serializer.serialize_str(&s)

The Serializer trait has other methods such as serialize_f32 that is what you are looking for:

mod compact_float {
    //! rounds a float to 3 decimal places, when serialized into a str, such as for JSON
    //! offsers space savings when such such precision is not needed.
    use serde::{Deserialize, Deserializer, Serializer};

    pub fn serialize<S>(float: &f32, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let s = format!("{:.3}", float);
        let parsed = s.parse::<f32>().unwrap();
        serializer.serialize_f32(parsed)
    }

    pub fn deserialize<'de, D>(deserializer: D) -> Result<f32, D::Error>
    where
        D: Deserializer<'de>,
    {
        f32::deserialize(deserializer)
    }
}