Use the ".text" and ".json" content of reqwest::blocking::Response multiple times

Hello everybody. I have been studying RUST for a couple of month, and just when I though I kind of understood the main concept, I stumbled on this problem.

TLDR: need to access the content of Response multiple times, is there a way without taking ownership? Recall that Request does not support clone().

I am using reqwest to obtain some data which content can be in either JSON or TSV.
Those information are wrapped in a reqwest::blocking::Response.

My intention is to use the content of the Request multiple times.
For example, I might just want to dump the raw content into a file for use in the future,
and at the same time I want to process the content (e.g. using serde_json to "extract" the json data).

The problem is two-folds:

  1. reqwest::blocking::Response cannot be cloned
  2. to obtain the text (for TSV) or to serialize to JSON I require to perfom an unwrap() which takes ownership.

How can I reuse the content of Request?

Following is (a simplified version of the code):

Please, note the following:

  • Response does not support clone() (to my understanding)
  • In the real code I also use Request.text() which requires unwrap()
  • Dumping the file also requires unwrap() regardless of the response format
use std::path::PathBuf;
use std::fs::File;
use reqwest::{blocking::Client, blocking::Response};


/// store the raw proteome meta-data as it was received
fn dump_raw_response(http_response_content: Response, outdir: PathBuf) {
    let outpath = outdir.join(format!("response.json"));
    let mut ofd = File::create(&outpath).unwrap_or_else(|error| {
        panic!("Could not create the file {:?}", error);
    });

    // Deserialize and write to file
    let json: serde_json::Value = http_response_content.json().unwrap();
    serde_json::to_writer_pretty(ofd, &json).unwrap();

    println!("{:#?}", outpath);

}


fn main() {

    let query_url = "https://rest.uniprot.org/proteomes/stream?compressed=false&download=false&fields=upid,organism,organism_id,protein_count,busco,cpd,genome_assembly,genome_representation&format=json&query=(organism_id:9606)+AND+(proteome_type:all)";
    let outdir = PathBuf::from("/my/output/dir/");

    // build the client
    let http_client = Client::new();
    // Perform the request
    let http_response = http_client.get(query_url).send();

    let http_response_content: Response;
    if http_response.is_err() {
        panic!("Error while retrieving data: {:#?}", http_response.err());
    }
    else {
        http_response_content = http_response.ok().unwrap();
    }

    // At this point http_resp_content contain the Response
    // which I would like to use multiple times

    dump_raw_response(http_response_content, outdir);

    // NOTE: calling json would take ownership
    let mut json: serde_json::Value = http_response_content.json().unwrap();
    // This json data get further processed...
    println!("{}", json);

I have tried:

  • passing the Response as reference, but I simply get a different error

Thanks a lot for all your help, and sorry for the lengthy post.

The method you're looking for is bytes. The returned Bytes value can be used however you want, like to convert to Vec or deserialize with serde_json::from_slice.

Response isn't actually the whole response. It's just the response headers and an open network connection. Calling bytes, json, or text passes the response content directly into a Bytes, serde type, or String respectively.

2 Likes

Thanks a lot @drewtato , that seems to do the trick!

Response isn't actually the whole response. It's just the response headers and an open network connection.

I did not know that, and after I read it my errors made more sense.

I also did not know the bytes crate!
It seems to give a lot of flexibility.

Thanks a lot again.

1 Like