[serde] Can "map.next_value()?" somehow return a raw json string instead of map?

help
1

Hello Community!

I would like to implement deserialization of arbitrary JSON messages using their type descriptor.
The solution works in this case.
But when I add the recursive nested JSON capability, I added a recursive call inside DesrializeSeed:

message_descriptor_instance.deserialize(
        &mut serde_json::Deserializer::from_str(map.next_value()?)
).unwrap()

and &mut serde_json::Deserializer::from_str(map.next_value()?) calls the error:

"invalid type: map, expected a borrowed string", line: 4, column: 30)

The error is correct and the card is in this field.
But do you know if I can get a raw json string instead of a map in this context?

Thanks!

use serde; // 1.0.130
use serde::de::DeserializeSeed;
use serde::de::Error;
use serde::de::MapAccess;
use serde::de::Visitor;
use serde::Deserializer;
use serde::{Deserialize, Serialize};
use serde_json; // 1.0.69
use std::collections::HashMap;

// Serde code will automatically map my value either to Int32 or Int64
#[derive(Debug, Serialize, Deserialize)]
#[serde(transparent)]
struct MessageValue {
    data: HashMap<String, Value>,
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(untagged)]
enum Value {
    Int32(i32),
    Int64(i64),
    Message(MessageValue)
}

// But I'd like to set value type based on message descriptor data, which are unknown at compilation time
#[derive(Debug, Serialize, Deserialize)]
#[serde(transparent)]
struct MessageDescriptor {
    data: HashMap<String, String>,
}
pub fn do_clone<K: Clone, V: Clone>(data: &HashMap<K,V>) -> HashMap<K, V> {
    data.clone()
}
impl MessageDescriptor {
    pub fn get(&self, key: &str) -> Option<&str>{
        self.data.get(key).map(|s| &**s)
    }
}

impl<'de> DeserializeSeed<'de> for MessageDescriptor {
    type Value = MessageValue;

    fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
    where
        D: Deserializer<'de>,
    {
        struct MyVisitor(MessageDescriptor);

        impl<'de> Visitor<'de> for MyVisitor {
            type Value = MessageValue;

            fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
                write!(formatter, "TODO")
            }

            fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
            where
                A: MapAccess<'de>,
            {
                let mut data: HashMap<String, Value> = HashMap::new();
                while let Some(key) = map.next_key::<String>()? {
                    let value: Value = match self.0.get(&key) {
                        Some("int64") => Value::Int64(map.next_value()?),
                        Some("int32") => Value::Int32(map.next_value()?),
                        Some("message") => {
                            let new_desc = MessageDescriptor{data: do_clone(&self.0.data)}; 
                            Value::Message(
                                new_desc.deserialize(
                                    &mut serde_json::Deserializer::from_str(
                                        map.next_value()?
                                    )
                                ).unwrap()
                            )
                        },
                        _ => {
                            return Err(A::Error::custom(
                                "MessageDescriptor does not know the type",
                            ))
                        }
                    };
                    data.insert(key, value);
                }
                Ok(MessageValue { data })
            }
        }
        deserializer.deserialize_map(MyVisitor(self))
    }
}

fn main() {
    // I would like to be able to provide some descriptor in runtime to deserialize the value to needed type.
    let msg_descriptor = r#"{
        "some_int32_field": "int32",
        "some_int64_field": "int64",
        "some_message_field": "message"
    }"#;
    let msg_descriptor_deserialized: MessageDescriptor =
        serde_json::from_str(msg_descriptor).unwrap();
    println!("msg_descriptor {:?}", msg_descriptor_deserialized);
    
    let msg = r#"{
        "some_int32_field": 23,
        "some_int64_field": 700,
        "some_message_field": {
            "some_int32_field": 23,
            "some_int64_field": 700
        }
    }"#;
    let _msg_deserialized: MessageValue = msg_descriptor_deserialized
        .deserialize(&mut serde_json::Deserializer::from_str(msg))
        .unwrap();
}

(Playground)

Output:

msg_descriptor MessageDescriptor { data: {"some_message_field": "message", "some_int32_field": "int32", "some_int64_field": "int64"} }

Errors:

   Compiling playground v0.0.1 (/playground)
    Finished dev [unoptimized + debuginfo] target(s) in 2.50s
     Running `target/debug/playground`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("invalid type: map, expected a borrowed string", line: 4, column: 30)', src/main.rs:111:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2

There is a tricky workaround, but it doesn't look very good I guess

If we replace

descriptor_instance.deserialize(
    &mut serde_json::Deserializer::from_str(&map.next_value())
).unwrap()

with

descriptor_instance.deserialize(
    &mut serde_json::Deserializer::from_str(
        &serde_json::to_string(
            &map.next_value::<HashMap<String,Value>>()?
        ).unwrap()
    )
).unwrap()

the script works correctly

1 Like
3

Rusts type inference will infer that this map.next_value()? must produce a &str. The actual value in the JSON however is a { ... } map, which cannot be deserialized as a &str.

You can directly pass a seed into the nested value, by calling next_value_seed instead of next_value.

Some("message") => {
    let new_desc = MessageDescriptor{data: do_clone(&self.0.data)};
    Value::Message(map.next_value_seed(new_desc)?)
},

You never modify the seed value. You might be able to change the DeserializeSeed impl to impl<'de> DeserializeSeed<'de> for &MessageDescriptor, i.e., implement it for the reference to MessageDescriptor. This might allow you to get rid of that clone.

If you ever need something like this, there is a RawValue type in serde_json, but here it is better solved by using next_value_seed.

2 Likes
4

Thank you for your help Jonas!
I really like the approach with next_value_seed!

@jonasbb maybe you have an idea on top of mind If we enrich Value with MessageRepeated what is the most elegant way to hadle sequence of Message's

#[derive(Debug, Serialize, Deserialize)]
#[serde(transparent)]
struct MessageValue {
    data: HashMap<String, Value>,
}

#[derive(Debug, Serialize, Deserialize)]
#[serde(untagged)]
enum Value {
    Int32(i32),
    Int64(i64),
    Message(MessageValue),
    MessageRepeated(Vec<MessageValue>)
}

let msg = r#"{
  "some_int32_field": 23,
  "some_message_field_repeated": [
    {
      "some_int32_field": 23,
      "some_int64_field": 700
    },
    {
      "some_int32_field": 23,
      "some_int64_field": 700
    }
  ]
}"#;

I have an idea to handle this as RawValue array:

Some("message_repeated") => {
    let new_codec = Codec {namespace: self.0.namespace, message: msg_desc};

    Value::MessageRepeated(
        map.next_value::<Vec<Box<RawValue>>>()?
            .iter()
            .map(|raw_json|
                new_codec.deserialize(
                    &mut serde_json::Deserializer::from_str(&raw_json.to_string())
                ).unwrap()
            )
            .collect()
    )
}

But I am trying to figure out what I need to do to do it with serde functions better

Thank you!

5

No, don't have a good idea. I guess the most "correct" way would be to create a DeserializeSeed implementation for the Vec which passes a seed for each element. But that solution requires a bit of code.

If your RawValue array works that is good too.
next_value::<Vec<Box<RawValue>>>() : You can change the inner type to Vec<&RawValue> and remove the box allocation.
&raw_json.to_string(): You can change this to raw_json.get(). to_string is tied to Display and unnecessarily creates a String first only to then deref into a &str.

1 Like
6

thank you for advice!
yes, rawvalue looks good

7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.