Read each line of an HTTP stream

[endless_flapjacks]

I'm using tokio and reqwest to make a request to a local HTTP server that sends back newline delimited data.

I'd like to stream the response, line by line as the entire response can be very large. I don't want to hold the entire response in memory.

My solution below works, but only by accident because the server sends each response line-by-line but I can't guarantee that's what will always occur.

use futures_util::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("starting");

    let mut stream = reqwest::get("http://localhost:8080")
        .await?
        .bytes_stream();

    while let Some(item) = stream.next().await { // <- stream.read_lines() would be great here!
        println!("Chunk: {:?}", item?);
    }
    Ok(())
}

I looked at tokio::io::Lines but the reqwest stream doesn't implement tokio::io::AsyncBufRead

How should I go about this? Any guidance would be greatly appreciated, thanks!

[Hyeonu]

To convert stream of bytes into AsyncBufRead:

StreamReader in tokio_util::io - Rust

[semicoleon]

If you don't have an upper bound on the size of the chunks, using tokio::io::BufReader to get an AsyncBufRead instance may not work for you reliably. You can implement this fairly directly with a Vec though.

use futures_util::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("starting");

    let mut stream = reqwest::get("http://localhost:8080/").await?.bytes_stream();

    let mut buffer = Vec::new();
    while let Some(item) = stream.next().await {
        for byte in item? {
            // Might need to consider carriage returns too depending
            // on how the server is expected to send the data.
            if byte == b'\n' {
                println!("Got chunk: {:?}", buffer.as_slice());
                buffer.clear();
            } else {
                buffer.push(byte);
            }
        }
    }

    Ok(())
}

There are probably ways to make that more efficient depending on the average expected chunk size, but it does work for me as is.

[endless_flapjacks]

Thanks @Hyeonu, that is the nudge in the direction I needed and I have it working.

I'll admit I don't entirely understand it but I can go through each part now and dig deeper.

[endless_flapjacks]

Thanks @semicoleon , I had considered something similar too but was hoping to avoid it. That was my fallback plan though, thanks for confirming it wasn't too crazy of an idea!