Extracting Hyper request.Body() into a String


#1

Hi all, I’m wondering if anyone can help me with a Hyper problem (which may just turn out to be an “I don’t understand a fundamental thing about Rust/Hyper” problem).

I’m trying to convert the request.Body() into a string. I’ve ended up with the code section below. The full script is in this gist.

    let body = req.body()
        .fold(Vec::new(), |mut acc, chunk| {
            acc.extend_from_slice(&*chunk);
            futures::future::ok::<_, Self::Error>(acc)
        })
        .and_then(|v| {
            println!("{:?}", v);
            let stringify = String::from_utf8(v).unwrap();
            println!("Anything in the string?: {}", stringify);
            Ok::<_, Self::Error>(stringify)
        });
    println!("Body: \n{:?}", body.wait().unwrap());

The output I get is:

    []
    Anything in the string?: 
    Body:

So I’m just ending up with an empty string! I know that there is something in the request.Body(), as I can extract it using a code example found from elsewhere (gist here for reference).

I’m sure that the issue is in the .fold() connector, but I just don’t know what it is. Can anyone help me with this?

Thanks!


#2

Hi, here’s some working example:

It seems that there’s nothing wrong with your code. Could you check if a server returns body as expected? Some sites does not return body on redirect (for example, http://rust-lang.org).

$ curl -sv 'http://rust-lang.org' | wc -c
* Rebuilt URL to: http://rust-lang.org/
*   Trying 192.241.171.49...
* Connected to rust-lang.org (192.241.171.49) port 80 (#0)
> GET / HTTP/1.1
> Host: rust-lang.org
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 05 Sep 2017 14:00:02 GMT
< Server: Apache
< Location: https://www.rust-lang.org
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host rust-lang.org left intact
0

#3

Hi,

thanks for the reply! Unfortunately I’m getting the same empty string via the concat2 method :frowning:

    let body = req.body().concat2().map_err(|_err| ()).map(|chunk| {
            let v = chunk.to_vec();
            String::from_utf8_lossy(&v).to_string()
        });

v ends up being an empty Vec, which makes sense, as println!("{:?}" chunk); also just returns [] .

In my implementation, I am sending data in a POST request from an HTML webform (gist of JS script here) to the Hyper server. This is all on a local Vagrant machine.

I have managed to extract the body from the same POST request using an alternative method (see previously linked gist). This was essentially copied from the Hyper Echo server tutorial.

But this is not the way I would like to manage the requests.

I don’t want to process the request body as part of the generation of the response body. I want to extract the request body, perform some kind of logic using it, then construct and return the response body.

Mainly though, I’d just like to understand why the request body appears to be present when processing it as part of the function to set the response body, but when trying to process the request body into a variable as part of the main function the fold() (or concat2) methods return empty Vecs…

Any further thoughts on this would be greatly appreciated. I feel like there is something really simple that I’m missing!

Thanks!


#4

I don’t see what drives that future (stream) to completion? You create the body chain and block on it (i.e. wait). But I’m not sure what completes it beyond the client connection terminating and probably all data dropped? I think you need to return a future from that function that incorporates your processing chain so that the event loop drives the stream. For example, something like:

let body = req.body()
            .fold(Vec::new(), |mut acc, chunk| {
                acc.extend_from_slice(&*chunk);
                futures::future::ok::<_, Self::Error>(acc)
            })
            .and_then(|v| {
                println!("{:?}", v);
                let stringify = String::from_utf8(v).unwrap();
                println!("Anything in the string?: {}", stringify);
                Ok::<_, Self::Error>(stringify)
            })
            .and_then(|_| {
                println!("Done");
                futures::future::ok(response.with_headers(headers))
            }).boxed();

        body
        //println!("Body: \n{:?}", body.wait().unwrap());

        /*let future = futures::future::ok(response.with_headers(headers));
        Box::new(future)*/

You don’t want to block (wait) in the event loop/async code.


#5

And here is the fundamental thing that I was missing! Thanks vitalyd, problem solved, and I think I understand better the workflow I need to use with Hyper / futures.


#6

Whilst the code within the call() function is now working exactly as expected, the call() function is triggering twice per request (I added a print statement to the top of the call() fn just to make sure). In the first request the request.body() is empty, whilst in the second request it is there.

Any ideas as to why this might be?


#7

Can you paste the contents of the call fn as you have it now?


#8
struct WebService;

impl Service for WebService {
    type Request = Request;
    type Response = Response;
    type Error = hyper::Error;
    type Future = Box<futures::Future<Item = Self::Response, Error = Self::Error>>;

    fn call(&self, req: Request) -> Self::Future {
        println!("FUNCTION CALLED?");
        let mut response = Response::new();
        let mut headers = Headers::new();
        let mut ContentType = Ascii::new("Content-Type".to_owned());
        let mut AccessControlAH = Ascii::new("Access-Control-Allow-Headers".to_owned());
        
        headers.set(AccessControlAllowHeaders(vec![ContentType, AccessControlAH]));
        headers.set(AccessControlAllowOrigin::Any);

        let body = req.body()
            .fold(Vec::new(), |mut acc, chunk| {
                acc.extend_from_slice(&*chunk);
                futures::future::ok::<_, Self::Error>(acc)
            })
            .and_then(|v| {
                let stringify = String::from_utf8(v).unwrap();
                Ok::<_, Self::Error>(stringify)
            })
            .and_then(|_| {
                futures::future::ok(response.with_headers(headers))
            }).boxed();

        body
    }
}

fn main() {
    let addr = "0.0.0.0:3000".parse().unwrap();
    let server = Http::new().bind(&addr, || Ok(WebService)).unwrap();
    server.run().unwrap();

#9

Hmm. Are you able to reproduce that via curl?


#10

Ah, interesting. No, using curl the function is only called once.

I just tested again, and the issue appears to be intermittent. The first time I click the button in the HTML form, the call() function triggers twice. Subsequently it triggers only once, except occasionally it triggers twice again!

So I get output that looks like this after multiple button presses:

FUNCTION CALLED?

FUNCTION CALLED?
Here's the body text!
FUNCTION CALLED?
Here's the body text!
FUNCTION CALLED?

FUNCTION CALLED?
Here's the body text!
FUNCTION CALLED?
Here's the body text!

I added a console.log to my JS, and I can see that the function that sends the data is only triggering once per button click.

Might this be something to do with buffering the request before reading the request.body()?


#11

Sorry, stupid question - where do you print the “Here’s the body text!” from? It’s not in the last snippet you gave.

Also, printing out the Request object may shed some light on what exactly is hitting your server: println!("{:?}", req);


#12

I added a print statement into the first .and_then connector:

.and_then(|v| {
            let stringify = String::from_utf8(v).unwrap();
            println!("{}", stringify);
            Ok::<_, Self::Error>(stringify)
        })

And you know what, I think printing the req object has revealed the issue. The method on the initial request is OPTIONS (HTTP docs here). This is the only difference between the two requests. So I think that the client is sending an initial request to establish the options available on the server, then subsequently sending the POST request.

So, one solution here is to match on the Method value of the request.

Thanks for all your help with this, I think I’m almost there.


#13

Ahh, ok - interesting indeed! :slight_smile: Good luck with the rest of the way.


#14

FYI this is the final script to extract the request.body() from Post requests as a String.

extern crate hyper;
extern crate futures;
extern crate unicase;

use hyper::Post;
use hyper::header::{Headers, AccessControlAllowOrigin, AccessControlAllowHeaders};
use hyper::server::{Http, Request, Response, Service};
use futures::Stream;
use futures::future::*;
use unicase::Ascii;
use std::str;

struct WebService;

impl Service for WebService {
    type Request = Request;
    type Response = Response;
    type Error = hyper::Error;
    type Future = Box<futures::Future<Item = Self::Response, Error = Self::Error>>;

    fn call(&self, req: Request) -> Self::Future {
        let (method, uri, _version, headers, body) = req.deconstruct();
        let response = Response::new();
        let mut headers = Headers::new();
        let content_type = Ascii::new("Content-Type".to_owned());
        let access_control_ah = Ascii::new("Access-Control-Allow-Headers".to_owned());
        
        headers.set(AccessControlAllowHeaders(vec![content_type, access_control_ah]));
        headers.set(AccessControlAllowOrigin::Any);
        match method {
            Post => {
                let body = body
                    .fold(Vec::new(), |mut acc, chunk| {
                        acc.extend_from_slice(&*chunk);
                        futures::future::ok::<_, Self::Error>(acc)
                    })
                    .and_then(|v| {
                        let stringify = String::from_utf8(v).unwrap();
                        println!("{}", stringify);
                        Ok::<_, Self::Error>(stringify)
                    })
                    .and_then(|_| {
                        futures::future::ok(response.with_headers(headers))
                    }).boxed();
                body
            },
            _ => Box::new(futures::future::ok(response.with_headers(headers)))
        }
    }
}

fn main() {
    let addr = "0.0.0.0:3000".parse().unwrap();
    let server = Http::new().bind(&addr, || Ok(WebService)).unwrap();
    server.run().unwrap();
}

#15

Cool. By the way, you can drop that body binding and the lonesome body expression in the end - just use the chain as the expression with the terminating boxed() call.

There’s also no real reason to schedule another and_then continuation to return the response - just do that from the 2nd and_then where you stringify the vec.


#16

Good shout.

extern crate hyper;
extern crate futures;
extern crate unicase;

use hyper::Post;
use hyper::header::{Headers, AccessControlAllowOrigin, AccessControlAllowHeaders};
use hyper::server::{Http, Request, Response, Service};
use futures::Stream;
use futures::future::*;
use unicase::Ascii;
use std::str;

struct WebService;

impl Service for WebService {
    type Request = Request;
    type Response = Response;
    type Error = hyper::Error;
    type Future = Box<futures::Future<Item = Self::Response, Error = Self::Error>>;

    fn call(&self, req: Request) -> Self::Future {
        let (method, uri, _version, headers, body) = req.deconstruct();
        let response = Response::new();
        let mut headers = Headers::new();
        let content_type = Ascii::new("Content-Type".to_owned());
        let access_control_ah = Ascii::new("Access-Control-Allow-Headers".to_owned());
        
        headers.set(AccessControlAllowHeaders(vec![content_type, access_control_ah]));
        headers.set(AccessControlAllowOrigin::Any);
        match method {
            Post => {
                body.fold(Vec::new(), |mut acc, chunk| {
                        acc.extend_from_slice(&*chunk);
                        futures::future::ok::<_, Self::Error>(acc)
                    })
                    .and_then(|v| {
                        let stringify = String::from_utf8(v).unwrap();
                        println!("{}", stringify);
                        futures::future::ok(response.with_headers(headers))
                    }).boxed()
            },
            _ => Box::new(futures::future::ok(response.with_headers(headers)))
        }
    }
}

fn main() {
    let addr = "0.0.0.0:3000".parse().unwrap();
    let server = Http::new().bind(&addr, || Ok(WebService)).unwrap();
    server.run().unwrap();
}

#17

And since you’re polishing/simplifying, may as well replace the fold with Stream::concat2 :slight_smile:.


#18

Is this what you were thinking?

body.concat2()
    .and_then(|body| {
        let vec = body.iter().cloned().collect(); 
        let stringify = String::from_utf8(vec).unwrap();
        println!("{}", stringify);
        futures::future::ok(response.with_headers(headers))
    }).boxed()

#19

Pretty much. I don’t think you need the cloned and collect calls though. body should be an “aggregate” Chunk that contains all the bytes of the request. As such, you can deref it to get a byte slice and pass it to String::from_utf8.


#20

Pow! String::from_utf8 expected a Vec, but str::from_utf8 accepted a slice.

req.body().concat2()
    .and_then(|body| {
        let stringify = str::from_utf8(&body).unwrap();
        println!("{}", stringify);
        futures::future::ok(response.with_headers(headers))
    }).boxed()