How to parse json using serde where target values are deeply nested?

I am trying to parse json response from an API into a struct using serde_json.
Here's a very simplified sample response

{
  "items": [
    {
      "label": {
        "volume": {
          "a": "...",
          "b": "..."
        }
      }
    },
    {
      "label": {
        "volume": {
          "a": "...",
          "b": "..."
        }
      }
    },
   ...
   ...
  ]
}

Response is rather dense but I am interested in very few values inside "volume" object.

struct Resp {
    a: AType,
    b: BType,
}

Parsing would be simple enough if I were to use derive attributes but unfortunately the values associated with "volume" are very chaotic and thus require manual deserialize implementation. (Although that was not as difficult as I'd imagined thanks to serde awesome docs!)

So basically what I am looking for is a way to point that manually written deserializer to the inner list structure and collect the parsed values into Vec<Resp>.


So far I've found two very hacky ways of going about that:

  1. Using serde_json::from_value which lead to some very hard to read code
serde_json::from_value::<Vec<Resp>>(
            reqwest::get(req)
                .await
                .map_err(...)?
                .json::<serde_json::Value>()
                .await
                .map_err(...)?["items"]
                .take(),
        ) // "items" is an array of maps.
        .map_err(...)?
        .iter_mut()
        .map(|v: &mut serde_json::Value| {
            serde_json::from_value(v["label"].take()).map_err(...)
        }) // Each map contains "label" key with a map value.
        ...
        ...
  1. Using intermediate structs with Deserialize attributes
#[derive(Deserialize)]
struct Items {
    items: Vec<Label>,
}

#[derive(Deserialize)]
struct Label{
    label: Volume,
}

#[derive(Deserialize)]
struct Label{
    volume: serde_json::Value,
}


serde_json::from_value::<Vec<Resp>>(
            reqwest::get(req)
                .await
                .map_err(...)?
                .json::<Items>()
                .await
                ...
                ...

which I think is essentially the same thing just more readable.

So I was wondering if there was a better way of targeting an inner structure with serde's deserialize fn.



P.S. I also tried volume: Resp in Label struct but that doesn't work, throws error like this reqwest::Error { kind: Decode, source: Error("expected ',' or '}' ...)

This should already return the thing you need, if I understand correctly (well, it returns reqwest::Result<Items>, obviously, but this is close), no need for from_value.

Yes it does but because I ultimately need a Vec<Resp>, Vec<Items> needs to be deconstructed which takes me back to iter().map(|...| ....map_err(...)) chain I mentioned in point 1 making code a bit hard to reason.

I was wondering if there was a way to avoid that and point deserializer implemented for Resp to each "volume" value and finally collect Vec<Resp> without these intermediate transforms.

You can use the with-attribute from serde to change how individual fields are de-/serialized. You can write a function which uses the intermediate structs from point 2 to deserialize the value, unwrap the values and return a Vec<Resp>. This way the intermediate structs are limited to the deserialization code.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=6613da35c61dc4c528e90d6b63db30e0

The example uses a simple deser_resp definition, which however collects an intermediate Vec. This can be avoided if necessary.

2 Likes

That was exactly what I was looking for and it works perfectly! Thank you

About the error I mentioned

reqwest::Error { kind: Decode, source: Error("expected ',' or '}' ...)

This was due to the way unknown value discard was implemented in visit_map's match block.

Field::A => { 
              // Parse A value into Option<A> 
            }

Field::B => { 
              // Parse B value into Option<B> 
            }
...
...
_        => {} // This one

This worked when the json was parsed using

serde_json::from_value

but not with

reqwest::get(req)
            .await
            .json::<Items>()
            ...

The fix was to use serde::de::IgnoredAny

Field::A => { 
              // Parse A value into Option<A> 
            }

Field::B => { 
              // Parse B value into Option<B> 
            }
...
...
 _ => {
           let _ = match V::next_value::<serde::de::IgnoredAny>(&mut map) {
               Ok(val) => val,
               Err(err) => {
                   return Err(err);
               }
           };
       }

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.