Build progress bar using surf

Hi,

I am trying to use surf http_client to read from an HTTP endpoint and want to read streams of bytes as and when available to build a progress bar. I am unable to find an utility that supports reading intermediate bytes. Can someone guide me on how to go about it?

Surf's Body type implements AsyncRead and AsyncBufRead, and you can get its length with .len(). So you will want to write code that gets the length of the body and then repeatedly calls fill_buf or read, updating the progress bar as more bytes are received.

1 Like

Somehow, I am unable to figure out the surf API to return a Stream for Responses, i.e. the body, on which I can use AsyncRead. It would be helpful if you can provide me with an example.

So you want to convert the Body to a stream of bytes? To do that you will need an adapter, you can use futures_codec::BytesCodec or asynchronous_codec::BytesCodec.

Sorry let me explain myself better.

So, currently I am using something like this -

let req = client.get("https://google.com").build();
            let mut resp = client.send(req).await;
            //println!("Response: {:?}", resp);
            let mut buf = Vec::<u8>::new();
            loop {
                let len = resp.read(&mut buf).await.unwrap();
                println!("Chunk Size: {}", len);
                if len == 0 {
                    break;
                }
         }

let mut resp = client.send(req).await; - returns the entire data at once.
I am new to async and struggling to understand how to make this return data as and when available. Let me know if this makes sense.

Are you sure it’s returning all the data at once? You currently have a loop to read the data which looks like it’s use AsyncRead, so I would think that that’s reading data as it comes in and not all at once. So you should be able to use the len variable each time through the loop to update a progress bar.

Yes, the len variable is printed as 0.
Furthermore,

//println!("Response: {:?}", resp); => prints the enitre body

which makes me believe that send( ) is not the right function I should call. According to my understanding, I am looking for something that returns a Stream (futures::stream::Stream - Rust) instead of Response

Sorry, ignore the above. The body in the response print was None.

let req = client.get("https://www.google.com/").build();
            let mut resp = client.send(req).await.unwrap();
            println!("Response: {:?}", resp);

            loop {
                let buf = resp.fill_buf().await.unwrap();

                let len = buf.len();
                println!("Chunk Size: {}", len);
                resp.consume_unpin(len);

                if len == 0 {
                    break;
                }
            }

This seems to be working for me now. Thanks :slight_smile:

Ah. Took me far longer than it should have to make me realize what’s going on. You need to pass read a buffer with a length to give it something to write into. If you pass it a Vec with length 0 it has nowhere to write data so it always returns 0. Typically that’s done with a 0 filled array [0 ; 8192] or a 0 filled Vec vec![0; 8192]. So if you set buf to one of those it should work. Such as

let mut buf = [0; 8192];
loop {
    let len = resp.read(&mut buf).await.unwrap();
    println!("Chunk Size: {}", len);
    if len == 0 {
        break;
    }
}

Alternatively if you want a stream of chunks of bytes, you can convert it to a Bytes object by doing something like this

let mut bytes_stream = resp.bytes();
while let Some(bytes) = bytes_stream.next().await {
    let len = bytes.len();
    ...
}
1 Like

Ah.. this makes sense! Thanks for the help.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.