Hyper::http POST only gets part of the body based on headers

help
#1

I wanted to make a quick post having spent a couple hours trying to diagnose odd behavior using the hyper::http client. I hope it will save someone some trouble in the future. But I do have a couple related questions to those with more experience in the tokio/hyper ecosystem.

I understand that the Response<Body> from a hyper::http request does not get sent all at once and may need to be buffered if you need the whole thing to deserialize into a struct etc. The below code (which I also placed in this repo) works using this approach described on SO:


use std::{env, fmt};
use serde::{self, Serialize, Deserialize, de::DeserializeOwned};
use serde_json;
use hyper::body; // brings the to_bytes() method into scope:
use hyper::{Request, Body, Method, Client};


// Make a GenericError and Generic Result that Box a bunch of statically-typed stuff
pub type GenericError = Box<dyn std::error::Error + Send + Sync>;
pub type GenericResult<T> = std::result::Result<T, GenericError>;


#[derive(Deserialize, Debug)]
struct Task {
    id: i32, 
    userId: i32,
    title: String,
    completed: bool,
}


#[derive(Serialize)]
struct Payload {
    userId: i32,
    title: String,
    body: String,
}

#[derive(Deserialize, Debug)]
struct SocialMediaPost {
    userId: i32,
    title: String,
    body: String,
    id: i32,
}


#[tokio::main]
async fn main() -> GenericResult<()> {

    // GET seems to buffer the whole body as expected
    let task: Task = get("http://jsonplaceholder.typicode.com/todos/10").await.unwrap();
    println!("{:?}", task); // WORKS: prints Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }

    
    // But why does POST have to have a header to get the whole thing back?
    let payload = Payload {
        userId: 1i32,
        title: "here are my thoughts".to_string(),
        body: "some idea here".to_string(),
    };
    let smp: SocialMediaPost = post("http://jsonplaceholder.typicode.com/posts", &payload).await.unwrap();
    println!("{:?}", smp);


    Ok(())
    
}

pub async fn get<T: DeserializeOwned>(url: &str) -> GenericResult<T> {
    let request = Request::builder()
        .method(Method::GET)
        .uri(url)
        .header("accept", "application/json")
        .body(Body::empty()).unwrap();
    let client = Client::new();
    let resp = client.request(request).await.unwrap();
    let bytes = body::to_bytes(resp.into_body()).await.unwrap();
    let foo = serde_json::from_slice::<T>(&bytes).unwrap();
    Ok(foo)

}


async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    let body_string = serde_json::to_string(payload).unwrap();
    let request = Request::builder()
        .method(Method::POST)
        .uri(url)
        .header("accept", "application/json")
        // IF YOU DON'T INCLUDE THIS HEADER, ONLY THE FIRST PROPERTY OF THE STRUCT GETS RETURNED???
        .header("Content-type", "application/json; charset=UTF-8")
        .body(Body::from(body_string)).unwrap();
    let client = Client::new();
    let resp = client.request(request).await.unwrap();
    let bytes = body::to_bytes(resp.into_body()).await.unwrap();
    println!("GOT BYTES: {}", std::str::from_utf8(&bytes).unwrap() );
    let foo = serde_json::from_slice::<T>(&bytes).unwrap();
    Ok(foo)

}

The GET request worked fine. However, the POST request only returned the first property of the return struct {id: 101} until I included the header .header("Content-type", "application/json; charset=UTF-8"). I do not understand why this would change the amount of data that was sent back? The header did not seem to be necessary when I make the same request with Postman.

Output from this code:

Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }
GOT BYTES: {
"userId": 1,
"title": "here are my thoughts",
"body": "some idea here",
"id": 101
}
SocialMediaPost { userId: 1, title: "here are my thoughts", body: "some idea here", id: 101 }

Output without the Content-type header

Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }
GOT BYTES: {
"id": 101
}
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error("missing field userId", line: 3, column: 1)', src/main.rs:90:51
note: run with RUST_BACKTRACE=1 environment variable

As one final question, the code above creates a Client::new() with each request. Is there any advantage to client re-use vs. this approach?

#2

It looks like the server is handling the request differently depending on the headers. Postman sets the Content-Type header automatically, so setting it in your Rust code is required to get the same results as Postman.

1 Like
#3

Why don't you use the reqwest crate instead? It's a convenient http client built on top of the hyper and written by the hyper maintainer. It's also recommended from hyper's document.

Carto.toml

[dependencies]
reqwest = { version = "0.11", features = ["json"] }

async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    let client = reqwest::Client::new();
    let req = client.post(url).json(payload);
    let resp = req.send().await?.json().await?;
    Ok(resp)
}

In case if you prefer one-liner:

async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    Ok(reqwest::Client::new().post(url).json(payload).send().await?.json().await?)
}
1 Like
#4

Thanks @Hyeonu ! That one-liner is elegant.

I considered using requwest, but as I was using Hyper for other parts of the project I wanted to keep everything in the same ecosystem as much as possible.

#5

Reqwest is part of the Hyper ecosystem. It's written by the author of Hyper, uses Hyper internally, and the Hyper docs recommend it for simple client-side use cases.

1 Like
#6

No I get it. I just started trying to do it directly in hyper and felt "d@$n it I'm so close- I can be a bit stubborn at times.

1 Like
#7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.