Get json data from site

help
1

Some site site.com returns json data, for example:

{
"a": {
    "text": ["abc", "xyz", "rty", "io", "f"], 
    "data": [
            [null, "txt", 18.1234, 12345678912345, -0.3]
            ]
     },
"b": {
    "text": ["qwe", "rty", "asd"], 
    "data": [
            [11343, 0, 20230407131355]
            ]
     }
}

I want to get it and then save in postgresql db as json record. Try http-client reqwest and serde:

[dependencies]
reqwest = { version = "0.11", features = ["json"] }
tokio = { version = "1", features = ["full"] }
serde = { version = "1.0.159", features = ["derive"] }
serde_json = "1.0"

main.rs:

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Debug)]
struct Msg {
    a: A,
    b: B,
}

#[derive(Serialize, Deserialize, Debug)]
struct A {
    text: Vec<String>,
    data: Vec<Vec<Option<Datum>>>,
}

#[derive(Serialize, Deserialize, Debug)]
struct B {
    text: Vec<String>,
    data: Vec<Vec<i64>>,
}

#[derive(Serialize, Deserialize, Debug)]
#[serde(untagged)]
enum Datum {
    Double(f64),
    String(String),
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
      // 
      let resp = reqwest::get("https://site.com")
                 .await?.json::<Msg>().await?;
      // 
      println!("{:?}", resp);   

Ok(())
}

i get something like this (println!):

Msg { a: A { text: ["abc", "xyz", ...

             data: [[None, Some(String("txt")), Some(Double(18.1234), Some(Double(12345678912345.0)), Some(Double(-0.3))]]

Want to get without A, Some(String / Double as origin.
How to do?

2

Serialize it to JSON with serde_json::to_string (serde_json - Rust), or get the response as text with reqwest.get("..").await?.text(). You can then turn that text into a Msg with serde_json::from_str and save the text to the database.

Rust's Debug format (which you get with println!("{:?}", resp), see Debug in std::fmt - Rust) is not intended for this purpose, just for printing and logging and such.

2 Likes
3

let r = serde_json::to_string(&resp)?;
how change 12345678912345.0 to 12345678912345

4

What is the fastest way to get and save in database as json original ? Thanks.

5

What for? It shouldn't make any difference to postgres, or anything else that accepts JSON, because JSON doesn't differentiate between integers and floats.

If I understood your question correctly, you can do something like this

let resp = reqwest::get("https://site.com").await?.text().await?;
let msg = serde_json::from_str::<Msg>(&resp)?;
store_json_to_database(resp);

If you wanted to do this, you should probably add #[serde(deny_unknown_fields)] on top of your Msg type to ensure that resp doesn't contain any extra data, as the Msg deserialization basically functions as validation in this case.

1 Like
6

I thought if the site gives json and I need to quickly accept and save it in db as json, then:
.await?.json::<Msg>().await?;
as native is fastest?

In both cases (.json or .text()) I get problem with my integer. 12345678912345.0 instead of 12345678912345

Position of the variables in my case is always the same, so need to edit the struct? How to view the result:
assert_eq!("{resp:?}"); ?
Thanks.

7

This is because you explicitly told Rust to deserialize it as a float, namely an f64:

If you want to be able to deserialize both floats and integers,
you could add a new variant to the enum, like this:

enum Datum {
   Integer(i64),
   Double(f64),
   String(String),
}

and it would always first try to deserialize it as an int (and then serialize too).

1 Like
8

I don't know what you mean by "as native", but if you just need to accept and save it, then .json::<Msg>() is going to do unnecessary extra work by deserializing the JSON into a struct.

I asked you why you need to change 12345678912345.0 to 12345678912345 but you didn't respond. What's the problem with 12345678912345.0?

Not sure what you mean by this.

1 Like
9

You are right, it`s works.
My sequence: integer after double

enum Datum {
	Double(f64),
	Integer(i64), // Integer(u64) 
	String(String),
}

didn't work and I deleted.
Thank you.

10

Yes, it made me understand.

println!()
distorts the result ?
give example code for my case, please
Thank you.

11

No, it's just unrelated to JSON. If you just want to see what kind of data resp holds, then println!("{:?}", resp); is useful. But it's not useful for anything else.

12

Good, thank you

13

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.