How to serialize vector of struct into a more compact json style?

help
1

Hello, as the title described. Assume we have the struct below:

use serde::ser::Serialize;

#[derive(Serialize)]
struct Data {
    id: usize,
    name: String,
    other: bool
}

The default serde_json will serialize Vec<Data> into string like below:

[
  {
    "id": 1,
    "name": "Jack",
    "other": true
  },
  {
    "id": 2,
    "name": "Michael",
    "other": true
  }
]

When the data grows, the result will contains many duplicate field names, and size growing.
How can I serialize each field into an array to get small size or better compression. Like below:

{
    "count": 3,
    "id": [1, 2, 3],
    "name": ["Jack", "Michael", "Tom"],
    "other": [true, true, false]
}

The result contains an additional field count to indicate the array length.

1 Like
2

This is known as the SoA vs AoS distinction. The easiest is probably to construct a struct with id, name and other being vectors and just serialize that.

Besides that, you can make your own DataVec and implement Serialize on it such that it serializes like that.

1 Like
3

Thanks for your reply!

I hope there is a generic way to do that, so could you please describe the second way with more details?

4

It would require manually implementing Serialize. There's a chapter about it in the serde manual: Implementing Serialize · Serde

1 Like
5

I've seen that doc, this maybe the fastest way right now. No generic way currently.
Thanks.

6

You can also write your own derive macro in order to make this approach generic.

7

The format you're proposing doesn't seem like a particularly good idea to me. It's quite difficult to read (for humans) and still pretty verbose.

If you want fewer bytes on the line but still keep the human-readability, run the JSON through ZIP or some other compression, that will transparently take care of the duplicate keys and is easily reversible.

If you don't care about human-readability, there are binary serialization formats that are much more compact than what you're proposing.

3 Likes
8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.