Hyper::http POST only gets part of the body based on headers

I wanted to make a quick post having spent a couple hours trying to diagnose odd behavior using the hyper::http client. I hope it will save someone some trouble in the future. But I do have a couple related questions to those with more experience in the tokio/hyper ecosystem.

I understand that the Response<Body> from a hyper::http request does not get sent all at once and may need to be buffered if you need the whole thing to deserialize into a struct etc. The below code (which I also placed in this repo) works using this approach described on SO:


use std::{env, fmt};
use serde::{self, Serialize, Deserialize, de::DeserializeOwned};
use serde_json;
use hyper::body; // brings the to_bytes() method into scope:
use hyper::{Request, Body, Method, Client};


// Make a GenericError and Generic Result that Box a bunch of statically-typed stuff
pub type GenericError = Box<dyn std::error::Error + Send + Sync>;
pub type GenericResult<T> = std::result::Result<T, GenericError>;


#[derive(Deserialize, Debug)]
struct Task {
    id: i32, 
    userId: i32,
    title: String,
    completed: bool,
}


#[derive(Serialize)]
struct Payload {
    userId: i32,
    title: String,
    body: String,
}

#[derive(Deserialize, Debug)]
struct SocialMediaPost {
    userId: i32,
    title: String,
    body: String,
    id: i32,
}


#[tokio::main]
async fn main() -> GenericResult<()> {

    // GET seems to buffer the whole body as expected
    let task: Task = get("http://jsonplaceholder.typicode.com/todos/10").await.unwrap();
    println!("{:?}", task); // WORKS: prints Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }

    
    // But why does POST have to have a header to get the whole thing back?
    let payload = Payload {
        userId: 1i32,
        title: "here are my thoughts".to_string(),
        body: "some idea here".to_string(),
    };
    let smp: SocialMediaPost = post("http://jsonplaceholder.typicode.com/posts", &payload).await.unwrap();
    println!("{:?}", smp);


    Ok(())
    
}

pub async fn get<T: DeserializeOwned>(url: &str) -> GenericResult<T> {
    let request = Request::builder()
        .method(Method::GET)
        .uri(url)
        .header("accept", "application/json")
        .body(Body::empty()).unwrap();
    let client = Client::new();
    let resp = client.request(request).await.unwrap();
    let bytes = body::to_bytes(resp.into_body()).await.unwrap();
    let foo = serde_json::from_slice::<T>(&bytes).unwrap();
    Ok(foo)

}


async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    let body_string = serde_json::to_string(payload).unwrap();
    let request = Request::builder()
        .method(Method::POST)
        .uri(url)
        .header("accept", "application/json")
        // IF YOU DON'T INCLUDE THIS HEADER, ONLY THE FIRST PROPERTY OF THE STRUCT GETS RETURNED???
        .header("Content-type", "application/json; charset=UTF-8")
        .body(Body::from(body_string)).unwrap();
    let client = Client::new();
    let resp = client.request(request).await.unwrap();
    let bytes = body::to_bytes(resp.into_body()).await.unwrap();
    println!("GOT BYTES: {}", std::str::from_utf8(&bytes).unwrap() );
    let foo = serde_json::from_slice::<T>(&bytes).unwrap();
    Ok(foo)

}

The GET request worked fine. However, the POST request only returned the first property of the return struct {id: 101} until I included the header .header("Content-type", "application/json; charset=UTF-8"). I do not understand why this would change the amount of data that was sent back? The header did not seem to be necessary when I make the same request with Postman.

Output from this code:

Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }
GOT BYTES: {
"userId": 1,
"title": "here are my thoughts",
"body": "some idea here",
"id": 101
}
SocialMediaPost { userId: 1, title: "here are my thoughts", body: "some idea here", id: 101 }

Output without the Content-type header

Task { id: 10, userId: 1, title: "illo est ratione doloremque quia maiores aut", completed: true }
GOT BYTES: {
"id": 101
}
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error("missing field userId", line: 3, column: 1)', src/main.rs:90:51
note: run with RUST_BACKTRACE=1 environment variable

As one final question, the code above creates a Client::new() with each request. Is there any advantage to client re-use vs. this approach?

It looks like the server is handling the request differently depending on the headers. Postman sets the Content-Type header automatically, so setting it in your Rust code is required to get the same results as Postman.

1 Like

Why don't you use the reqwest crate instead? It's a convenient http client built on top of the hyper and written by the hyper maintainer. It's also recommended from hyper's document.

Carto.toml

[dependencies]
reqwest = { version = "0.11", features = ["json"] }
async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    let client = reqwest::Client::new();
    let req = client.post(url).json(payload);
    let resp = req.send().await?.json().await?;
    Ok(resp)
}

In case if you prefer one-liner:

async fn post<U: Serialize, T: DeserializeOwned>(url: &str, payload: &U) -> GenericResult<T> {
    Ok(reqwest::Client::new().post(url).json(payload).send().await?.json().await?)
}
1 Like

Thanks @Hyeonu ! That one-liner is elegant.

I considered using requwest, but as I was using Hyper for other parts of the project I wanted to keep everything in the same ecosystem as much as possible.

Reqwest is part of the Hyper ecosystem. It's written by the author of Hyper, uses Hyper internally, and the Hyper docs recommend it for simple client-side use cases.

1 Like

No I get it. I just started trying to do it directly in hyper and felt "d@$n it I'm so close- I can be a bit stubborn at times.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.