Serialize a variable and very large JSON response

help
1

Hi,

I am trying to serialize a variable and very large JSON response (from a GET request from the zoomeye.org API) but without creating a struct as it would be very long and there is probably a better way that I ignore to do so...

Here is an example of the JSON I'm trying to serialize:

body: "{\"code\": 60000, \"total\": 126702344, \"available\": 126702344, \"matches\": [{\"rdns\": \"[34-92-16-210.limerick.co.in](https://34-92-16-210.limerick.co.in/)\", \"jarm\": \"\", \"ico\": {\"mmh3\": \"\", \"md5\": \"\"}, \"txtfile\": {\"robotsmd5\": \"\", \"securitymd5\": \"\"}, \"ip\": \"[210.16.92.34](https://210.16.92.34/)\", \"portinfo\": {\"hostname\": \"\", \"os\": \"Linux\", \"port\": 2223, \"service\": \"ssh\", \"title\": null, \"version\": \"\", \"device\": \"router\", \"extrainfo\": \"protocol 2.0\", \"rdns\": \"[34-92-16-210.limerick.co.in](https://34-92-16-210.limerick.co.in/)\", \"app\": \"MikroTik RouterOS sshd\", \"banner\": \"SSH-2.0-ROSSSH\\r\\n\"}, \"timestamp\": \"2022-12-16T20:26:41\", \"geoinfo\": {\"continent\": {\"code\": \"AP\", \"names\": {\"en\": \"Asia\", \"zh-CN\": \"亚洲\"}, \"geoname_id\": null}, \"country\": {\"code\": \"IN\", \"names\": {\"en\": \"India\", \"zh-CN\": \"印度\"}, \"geoname_id\": null}, \"base_station\": \"\", \"city\": {\"names\": {\"en\": \"Pune\", \"zh-CN\": \"浦那\"}, \"geoname_id\": null}, \"isp\": \"[veloxr.in](https://veloxr.in/)\", \"organization\": \"Limerick Technologies Pvt Ltd\", \"idc\": \"\", \"location\": {\"lon\": \"73.85218\", \"lat\": \"18.53611\"}, \"aso\": null, \"asn\": \"138311\", \"subdivisions\": {\"names\": {\"en\": \"Maharashtra\", \"zh-CN\": \"马哈拉施特拉邦\"}, \"geoname_id\": null}, \"PoweredBy\": \"IPIP\", \"scene\": {\"en\": \"Used\", \"cn\": \"已使用\"}, \"organization_CN\": null}, \"protocol\": {\"application\": \"HTTP\", \"probe\": \"GetRequest\", \"transport\": \"tcp\"}, \"honeypot\": 0, \"whois\": {\"2016-10-17\": {\"admin_c\": \"RG570-AP\", \"country\": \"IN\", \"descr\": \"SHREE BALAJI INFOWAY PRIVATE LIMITED\", \"inetnum\": \"[210.16.92.0](https://210.16.92.0/) - [210.16.95.255](https://210.16.95.255/)\", \"ip_end\": \"[210.16.95.255](https://210.16.95.255/)\", \"ip_start\": \"[210.16.92.0](https://210.16.92.0/)\", \"irt\": [{\"abus_mailbox\": \"[X](mailto:rajeshnaikare@gmail.com)XX", \"address\": \"SR. NO. 2/4, SHOP NO. 4, ADARSH NAGAR, DHANKAWADI, PUNE - 411043\", \"admin_c\": \"RG570-AP\", \"auth\": \"# Filtered\", \"email\": \XXX", \"fax_no\": \"\", \"irt\": \"IRT-IN-SHREEBALAJI\", \"irt_nfy\": \"\", \"last_modified\": \"2016-10-18T06:43:03Z\", \"mnt_by\": \"MAINT-IN-SHREEBALAJI\", \"notify\": \"\", \"phone\": \"\", \"source\": \"APNIC\", \"tech_c\": \"RG570-AP\"}], \"last_modified\": \"2016-10-17T07:16:50Z\", \"mnt_by\": \"MAINT-IN-IRINN\", \"mnt_irt\": \"IRT-IN-SHREEBALAJI\", \"mnt_lower\": \"\", \"mnt_routes\": \"MAINT-IN-SHREEBALAJI\", \"netname\": \"SHREEBALAJI\", \"notify\": \"\", \"organization\": null, \"person\": [{\"abuse_mailbox\": \"\", \"address\": \"SR. NO. 2/4, SHOP NO. 4, ADARSH NAGAR, DHANKAWADI, PUNE - 411043\", \"country\": \"IN\", \"email\": \"[X](mailto:rajeshnaikare@gmail.com)XX\", \"fax_no\": \"\", \"last_modified\": \"2016-10-18T06:42:18Z\", \"mnt_by\": \"MAINT-IN-SHREEBALAJI\", \"nic_hdl\": \"RG570-AP\", \"notify\": \"\", \"person\": \"RAJESH GAJANAN\", \"phone\": \"+91 9049994949\", \"source\": \"APNIC\"}], \"role\": [], \"source\": \"APNIC\", \"status\": \"ALLOCATED PORTABLE\", \"tech_c\": \"RG570-AP\"}}}, {\"jarm\": \"\", \"ico\": {\"mmh3\": \"\", \"md5\": \"\"}, \"txtfile\": {\"robotsmd5\": \"\", \"securitymd5\": \"\"}, \"ip\": \"[223.26.31.188](https://223.26.31.188/)\", \"portinfo\": {\"hostname\": \"\", \"os\": \"\", \"port\": 2222, \"service\": \"ssh\", \"title\": null, \"version\": \"4.3\", \"device\": \"\", \"extrainfo\": \"protocol 2.0\", \"rdns\": \"\", \"app\": \"OpenSSH\", \"banner\": \"SSH-2.0-OpenSSH_4.3\\n\"}, \"timestamp\": \"2022-12-16T20:26:41\", \"geoinfo\": {\"continent\":... //That is not even 1% of it...

Surely you understand why I can't just create a struct that I'd derive with Serialize and Deserialize for this purpose...

Thanks in advance,

Noah

2

The general approach is to either create a struct to de-serialize into or to de-serialize this as a JSON value, and then manually pick out the keys you want.
If you want a smarter approach, then you'd need to say what keys you need. There may be cleverer ways for specific cases, but there are none for the generic case.

3

Well here what I am trying to do is getting a list of target with these properties: IP, Service (SSH, FTP, etc...) and Port. I am not sure how to "manually pick out the keys" I want, could you explain a little bit more the way you'd approach that?

Thanks :wink:

Noah

4

JSON is notation for, what is effectively, a dictionary.
So if you have some JSON like:

{
  foo: {
    bar: [
      {
        baz: "....",
        ...
      },
      ....
    ]
    ....
}

If you want to find every value corresponding to the baz key, then you need to:

  1. Find the JSON value for key foo
  2. Find the JSON value (which is an array) for the key bar in the previous value
  3. Iterate over the previous value
  4. For each iterated value, find the value for the key baz.

This is what I meant by "manually" picking out the values.

5

IIUC you need to create a custom deserializer to do what you want. See Array of values without buffering · Serde for an example.

1 Like
6

Read about the Value enum in serde_json to understand what RedDocMD said: Value in serde_json::value - Rust

7

Do you really need to serialize (convert from data structure to JSON) this data, or only deserialize it? If you are just deserializing it, then you can declare a struct which contains only the fields you care about. Serde will ignore all other keys in the JSON.

4 Likes
8

I just found that out, Serde is magic!! ta mate!