Reading bytes into a buffer with Tokio

help
1

I'm looking at this example from Tokio: tokio/echo.rs at master · tokio-rs/tokio · GitHub

let mut buf = [0; 1024];

// In a loop, read data from the socket and write the data back.
loop {
    let n = socket
        .read(&mut buf)
        .await
        .expect("failed to read data from socket");

    if n == 0 {
        return;
    }
}

However, in my case it's possible that another side of the TCP connection is sending me more than 1024 bytes.

I'd like to do read in a loop until there's no more bytes to read.

I've tried read_to_end from AsyncReadExt which reads until EOF, which only works for files but not for the network.

I've seen this example in another code base:

impl ConnReader {
    fn new(stream: Rc<TcpStream>) -> Self {
        ConnReader {
            stream: stream,
            packet_buf: Vec::with_capacity(4096),
            read_buf: vec![0_u8; 4096]
        }
    }

    fn read(&mut self) -> Poll<(), io::Error> {
      loop {
          match self.stream.poll_read() {
              Async::Ready(_) => {
                  let n = try_nb!((&*self.stream).read(&mut self.read_buf[..]));
                  if n == 0 {
                      return Err(Error::new(ErrorKind::Other, "connection closed"));
                  }
                  self.packet_buf.extend_from_slice(&self.read_buf[0..n]);
              },
              _ => return Ok(Async::NotReady),
          }
      }
    }

This is very similar to what I'm looking for: read in a loop using the buf and concat results into a vector.

The problem is that the example is from 2016, it doesn't use any async/await and it's based on tokio 0.1.0 and none of that works in modern libraries.

I'd like to hear any suggestions for how to read bytes from the pipe in a loop into a vector, assuming I'm to read more bytes than my buffer allows.

Thanks

2

What do you mean by "until there's no more bytes to read"? read_to_end works just fine for the network: it reads until the other side of the connection shuts it down to indicate they've finished writing everything. TCP is a stream protocol - you'll generally need to implement some kind of message framing to know when the other side is finished writing if you want to communicate back and forth.

The example you referenced expects to read exactly 4,096 bytes. In modern tokio you can just use read_exact if you want to do that.

3

Perhaps you have a misunderstanding of what EOF means for tcp connections? EOF means the connection has been closed, not just that the other end has temporarily stopped sending more data.

4

thanks sfackler and alice.

TCP is a stream protocol - you'll generally need to implement some kind of message framing to know when the other side is finished writing if you want to communicate back and forth.

I see! My understanding of how TCP streams work wasn't very complete.

In my case it's the myself protocol, so I guess I'll have to read the first 3 bytes that indicate the length of the total packet. And then call read_exact with the buffer size equal to the expected length.