Curl POST + TcpStream

Guys, I'm kinda stuck with a problem here. I'm trying to understand the the basics of TCP server programming at a relatively low level, and I don't know whether my Rust code is wrong or curl doesn't do what I'm expectiong it to do.

On the curl side it's all simple:

curl --data "parm=value" localhost:33000

I tried all possible combinations and methods of reading from a TcpStream, including BufReader.


Both sides are blocked. When I CTRL+C curl, only then I get my buf ready.
Ok so read_to_end says it reads until EOF. I suspect curl doesn't send EOF with POST request?

let mut buf= vec![];
stream.read_to_end(&mut buf);  //blocks

In this case read successfully get called several times, but the last call is blocked and never returns.

let mut read_buf = [0u8; 32];
let mut data = vec![];
   loop {
        match reader.read(&mut read_buf) {
            Ok(0) =>  break, // this arm never called
            Ok(n) =>  data.extend_from_slice(&read_buf[..n]),
            Err(e) =>  eprintln!("{}", e),
        }
    }


Same as in previous case, read_line blocks right before the end.

let mut reader  = BufReader::new(stream);
let mut s = String::new();
loop {
    match reader.read_line(&mut s) {
        Ok(0) => {
            println!("Read zero bytes"); // arm never reached
            break;
        }
        Ok(n) => {
            println!("Read {}", n);
        }
        _ => ()
    }
}


Finally, this works and I don't know why and I don't feel this is a right solution
First, read doesn't guarantied to read all the data in one call, second the buf might not be large enough. But why doesn't this block?. With large enough buffer it somehow knows when to stop reading!

let mut buf = [0u8; 1024];
socket.read(&mut buf)

How do I properly read the request (not parse, just read the bytes)?

Hi,

Without reading the network frames it seems difficult to give an answer. On the other hand I find strange to use curl and TcpStream, it doesn't work on the same layer levels, right?
The test could be done with tools like ::

netcat

or a direct send like

dd if=/dev/zero bs=9000 count=1000 > /dev/tcp/$target_host/$port

read() will block until there are some data in the stream. When some data is available, it will write some of the data (or all of it) to the buffer and return.

read_to_end() will block until it encounters EOF. In the case of a TCP stream, EOF means the connection was terminated. Curl will not terminate the connection because it's waiting for an HTTP response, so read_to_end() will not return until curl is killed or its timeout is reached (if there is one).

When you call read() in a loop, you will receive all the data that curl sent. After that, the next call will block forever because curl is not going to send any more data.

read_line() will behave in a similar way. You will get the incoming data line by line. However, you won't get the final piece of data ("parm=value") because there is no line break after it. After receiving the incomplete line, read_line() will call read() again, and it will block.

The final ("working") version is not blocking because you're only calling it once. The problem with it is that you're not guaranteed to get all of the data. Even if the buffer is large enough, read() may return only the first part of the data (this will always happen once the request is big enough). To get more data, you have to call read() again, and that can block.

Blocking is not itself an issue. For example, you can use set_nonblocking to turn off it, and read() will just return a WouldBlock error instantly when there is no more data. However, that doesn't help to determine if you're expecting any more data in this request.

So, how to properly read the request, then? Well, the problem is that TCP doesn't have a concept of request. It's a stream. TCP doesn't provide a way to indicate that a request has ended (except closing a connection). This is what higher level protocols are for. In the case of HTTP, the header is line-separated, so you can use read_line() to read that. The last line will be empty (just a \r\n), informing you that you shouldn't be reading any more lines. If the request has a body, the Content-Length header will indicate its length, and you can use, for example, read_exact() to read the following data.

Some protocols are simpler. For example, length delimited encoding is a simple way to separate requests or messages in a stream. (The link is just for the description of the encoding. You don't need to use tokio if you're just trying out std's TCP API.)

@Riateche, thank you,

Ok read_to_end won't work because there's no EOF in the stream, now I get it.

This is what higher level protocols are for. In the case of HTTP, the header is line-separated, so you can use read_line() to read that. The last line will be empty (just a \r\n ), informing you that you shouldn't be reading any more lines. If the request has a body, the Content-Length header will indicate its length, and you can use, for example, read_exact() to read the following data.

This! I'm now able to detect \r\n with read_line.
Still curious however how to approach this properly. I'm guessing reading one byte at a time and parse ti.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.