Serde - HashMap serialization - key error

extern crate serde_derive;
extern crate serde;
extern crate serde_json;

use serde_derive::{Serialize, Deserialize};
use std::collections::HashMap;

#[derive(Serialize, Deserialize, Debug, Clone, Default, Hash,
 PartialEq, PartialOrd, Eq)]
pub struct Request
{
    pub offset: u16,
    pub count: u16
}

#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes { pub attr: HashMap<Request, u32> }

fn main() {
    let mut attrs = Attributes{ attr: HashMap::new() };
    attrs.attr.insert(Request{count: 1, offset: 2}, 42);
    let j = serde_json::to_string(&attrs).unwrap();
    println!("json: {:#?}", j);
}

Error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err`
value: Error("key must be a string", line: 0, column: 0)',
src/main.rs:26:43

How to fix it?

If you check out: https://docs.serde.rs/serde_json/fn.to_string.html

It says:

Serialization can fail if T 's implementation of Serialize decides to fail, or if T contains a map with non-string keys.

And hence, you're getting the error you see since the fields of Request are non-String values.

Check out: https://github.com/serde-rs/json/issues/402

The underlying problem is that JSON itself doesn't support non-string keys.

{
    { "offset": 2, "count": 1 }: 42
}

is simply not valid JSON.

Something you can do is write a custom serialize/deserialize implementation for Request that works with strings - here is an example of this which serializes requests as {offset}-{count}, but you can do anything there.

2 Likes

Instead of manually implementing the traits, like @Koxiaet suggested, you can see if one of the types in serde_with will help you.

You could use serde_with::json::JsonString:

#[serde_as]
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes {
    #[serde_as(as = "HashMap<serde_with::json::JsonString, _>")]
    pub attr: HashMap<Request, u32>,
}
{"attr":{"{\"offset\":2,\"count\":1}":42}}

You could also serialize the HashMap as a list of tuples:

#[serde_as]
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes {
    #[serde_as(as = "Vec<(_, _)>")]
    pub attr: HashMap<Request, u32>,
}
{"attr":[[{"offset":2,"count":1},42]]}

If your Request struct implements Display and FromStr then serde_with::DeserializeFromStr and serde_with::SerializeDisplay will come in handy.

2 Likes

@jonasbb, I cannot compile a simple example for serde_with.

use serde_with::{serde_as, DisplayFromStr};
use std::collections::HashMap;

#[serde_as]
#[derive(Serialize, Deserialize)]
struct Data {
    /// Serialize into number
    #[serde_as(as = "_")]
    a: u32,

    /// Serialize into String
    #[serde_as(as = "DisplayFromStr")]
    b: u32,

    /// Serialize into a map from String to String
    #[serde_as(as = "HashMap<DisplayFromStr, _>")]
    c: Vec<(u32, String)>,
}

fn main()
{
}
error[E0433]: failed to resolve: could not find `Deserialize` in `serde`
   --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/de/impls.rs:638:25
    |
638 |         #[derive(serde::Deserialize)]
    |                         ^^^^^^^^^^^ could not find `Deserialize` in `serde`

error[E0433]: failed to resolve: could not find `Deserialize` in `serde`
    --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/rust.rs:1707:32
     |
1707 |         #[derive(Debug, serde::Deserialize)]
     |                                ^^^^^^^^^^^ could not find `Deserialize` in `serde`

error: cannot find attribute `serde` in this scope
   --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/de/impls.rs:639:11
    |
639 |         #[serde(
    |           ^^^^^

error: cannot find attribute `serde` in this scope
    --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/rust.rs:1708:11
     |
1708 |         #[serde(untagged)]
     |           ^^^^^

error[E0277]: the trait bound `de::impls::<impl de::DeserializeAs<'de, T> for DefaultOnError<TAs>>::deserialize_as::GoodOrError<'_, T, TAs>: serde::Deserialize<'_>` is not satisfied
   --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/de/impls.rs:655:18
    |
655 |         Ok(match Deserialize::deserialize(deserializer) {
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `serde::Deserialize<'_>` is not implemented for `de::impls::<impl de::DeserializeAs<'de, T> for DefaultOnError<TAs>>::deserialize_as::GoodOrError<'_, T, TAs>`
    | 
   ::: /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.116/src/de/mod.rs:540:12
    |
540 |         D: Deserializer<'de>;
    |            ----------------- required by this bound in `serde::Deserialize::deserialize`

error[E0277]: the trait bound `default_on_error::deserialize::GoodOrError<_>: serde::Deserialize<'_>` is not satisfied
    --> /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_with-1.5.0/src/rust.rs:1717:18
     |
1717 |         Ok(match Deserialize::deserialize(deserializer) {
     |                  ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `serde::Deserialize<'_>` is not implemented for `default_on_error::deserialize::GoodOrError<_>`
     | 
    ::: /home/mhanusek/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.116/src/de/mod.rs:540:12
     |
540  |         D: Deserializer<'de>;
     |            ----------------- required by this bound in `serde::Deserialize::deserialize`

error: aborting due to 6 previous errors

Thanks, I will look into it. 99% sure it is cargo's incorrect feature resolution for dev-dependencies.
Try enabling the derive feature of serde.

serde = {version = "...", features = ["derive"]}

EDIT: Version v1.5.1 fixes the wrong feature in the serde dependency.

use std::fmt::{Formatter, Display};

use serde_derive::{Serialize, Deserialize};
use serde::{Serialize, Serializer};
use serde::ser::SerializeMap;

use std::collections::HashMap;

#[derive(Serialize, Deserialize, Debug, Clone, Default, Hash, PartialEq, PartialOrd, Eq)]
pub struct Request
{
    pub offset: u16,
    pub count: u16
}

impl Display for Request {
    fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
        write!(f, "offset: {}, count: {}", self.offset, self.count)
    }
}

#[derive(Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes
{
    pub attr: HashMap<Request, u32>
}

impl Serialize for Attributes
{
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where S: Serializer,
    {
        let mut map = serializer.serialize_map(Some(self.attr.len()))?;
        for (k, v) in &self.attr
        {
            map.serialize_entry(&k.to_string(), &v)?;
        }
        map.end()
    }
}

fn main()
{
    // not working
    // let mut z : HashMap<Request, u32> = HashMap::new();
    // z.insert(Request{count: 323, offset: 299}, 77);
    // let j = serde_json::to_string_pretty(&z).unwrap();
    // println!("json: {}", j);

    // working
    let mut attrs = Attributes{ attr: HashMap::new() };
    attrs.attr.insert(Request{count: 1, offset: 2}, 42);
    attrs.attr.insert(Request{count: 23, offset: 99}, 11);
    let j = serde_json::to_string_pretty(&attrs).unwrap();
    println!("json: {}", j);
}

How serialize a HashMap<Request, u32> ?

serde_with looks very nice, but if you're having trouble getting it to work correctly, you can always write the serialize and deserialize adapters manually. It's not a ton of work to round-trip the HashMap through Vec<(Request, u32)>. You could go through HashMap<String, u32> almost as easily providing whatever (de)serialization you want for Request.

#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes {
    #[serde(with = "attr")]
    pub attr: HashMap<Request, u32>,
}

mod attr {
    use serde::{Deserializer, Serializer};

    type Attr = std::collections::HashMap<super::Request, u32>;

    pub(super) fn serialize<S: Serializer>(attr: &Attr, ser: S) -> Result<S::Ok, S::Error> {
        let attr: Vec<_> = attr.iter().collect();
        serde::Serialize::serialize(&attr, ser)
    }

    pub(super) fn deserialize<'de, D: Deserializer<'de>>(des: D) -> Result<Attr, D::Error> {
        let attr: Vec<_> = serde::Deserialize::deserialize(des)?;
        Ok(attr.into_iter().collect())
    }
}
1 Like

@trentj, thanks for reply.

I don't understand why it is impossible to serialize HashMap?

pub(super) fn serialize<S: Serializer>(attr: &Attr, serializer: S) 
-> Result<S::Ok, S::Error>
{
   let mut map = serializer.serialize_map(Some(attr.len()))?;
   for (k, v) in attr.iter()
   {
     let key = format!("{{offset: {}, count: {}}}", k.offset, k.count);
     map.serialize_entry(&key, &v)?;
   }
   map.end()
}

How implement a deserialize function?

It's not impossible to serialize a HashMap, it's just not possible to serialize it as a JSON object with non-string keys, because JSON objects only have string keys. So the way that serde_json deals with this is by panicking when passed a HashMap whose keys are not stringy.

     let key = format!("{{offset: {}, count: {}}}", k.offset, k.count);

Ok, so this is the format you've chosen to stringify Request. Have you also written a function that turns a string like {offset: 10, count: 20} back into a Request? You'll need to use that in the deserialize function. (Or you could just use serde_json::to_string and serde_json::from_str, which do the hard work for you.)

The right way is to write a struct and implement serde::de::Visitor for it such that calling visit_map constructs the HashMap<Request, u32>.

The easy way is to just deserialize to a HashMap<String, u32> and convert it to a HashMap<Request, u32> after the fact:

    pub(super) fn deserialize<'de, D: Deserializer<'de>>(des: D) -> Result<Attr, D::Error> {
        let attr: HashMap<String, u32> = serde::Deserialize::deserialize(des)?;
        Ok(attr
            .into_iter()
            .map(|(k, v)| {
                (
                    // assuming the serialize function calls serde_json::to_string
                    serde_json::from_str(&k).expect("Could not deserialize Request"),
                    v,
                )
            })
            .collect())
    }

This is wasteful of space and probably also slower than doing it the "right" way. But it gets you moving (and if serialization is a bottleneck you probably shouldn't be using JSON).

I'd like to assume that serde_with does things the "right" way. If that's the case I'd probably use that instead. IIUC this is equivalent to @jonasbb's first suggestion:

#[serde_as]
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
struct Attributes {
    #[serde_as(as = "HashMap<serde_with::json::JsonString, _>")]
    pub attr: HashMap<Request, u32>,
}
1 Like