How to close a TcpStream?

[ache]

Hi \o,

I'm trying to send a file (HTTP) on the TCP level. So basically, it's a low-level static server.
I download it using wget nothing fancy. Usually, it works well. But sometimes, it just doesn't. wget say "Connection reset by peer" shortly before the end.

Here is what wget shown

$ env LC_ALL=C wget localhost:8080/big # LC_ALL => Set env to english
--2019-09-24 17:21:09--  http://localhost:8080/big
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... failed: Connection refused. # I'm not listening on ipv6
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.               # but on ipv4
HTTP request sent, awaiting response... 200 OK
Length: 1831366851 (1.7G) [application/octet-stream]
Saving to: 'big'

big                                      99%[============================================================================> ]   1.70G   570MB/s    in 3.1s    

2019-09-24 17:21:12 (570 MB/s) - Read error at byte 1828713146/1831366851 (Connection reset by peer). Retrying.

It will retry and eventually it will work.

I studied a little and it seems that "Connection reset by peer" mean that the socket receive a TCP "RST". My code doesn't seem to send a RST so I assume that it's send by the stream when Dropped (Drop trait).

To verify it, I use a short delay before dropping the Stream and it works very well. But that's not how i'm suppose to close a socket ! I just want the normal process FIN ACK FIN ACK.

I tried shutdown doesn't work. Using wireshark is hard because the file is big, will try with a medium sized file.

Can you help me to close that socket correctly ? I just want a « active close » to certify that the client receive everything.

Here is the code :

use std::{io, ptr};
use std::os::unix::io::AsRawFd;
use std::fs::File;
use std::net::TcpListener;
use std::io::Write;
use std::convert::TryInto;
use std::{thread, time};


pub fn send_file<R: AsRawFd, W: AsRawFd>(r: &mut R, w: &mut W, size: usize) -> io::Result<usize> {
    match unsafe { libc::sendfile(w.as_raw_fd(), r.as_raw_fd(), ptr::null_mut(), size) } {
        -1 => Err(io::Error::last_os_error()),
        copied => Ok(copied as usize),
    }
}

fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:8080")?;
    let delay = false; // Set to true to add a short delay
    let short_time = time::Duration::from_millis(500);

    // accept connections and process them serially
    for stream in listener.incoming() {
        let mut stream = stream.unwrap();
        stream.set_nodelay(false)?;
        let http_header_first = "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\nContent-length: ";
        let http_header_last = "\r\nServer: testSRV/0.1\r\nAccept-Ranges: bytes\r\n\r\n";

        let mut file = File::open("big")?;
        let metadata = file.metadata()?;
        let size : usize = metadata.len().try_into().unwrap();
        let http_header = http_header_first.to_owned() + &size.to_string() + http_header_last;

        std::println!("File of {} bytes", size);
        stream.write(http_header.as_bytes())?;
        let mut total_send = 0;
        let packet_size : usize = 2<<15;
        println!("Packet size: {}", packet_size);
        loop {
            let this_packet_size = if total_send + packet_size > size { size - total_send } else { packet_size };
            let i = send_file(&mut file, &mut stream, this_packet_size);
            match i {
                Ok(_)  => (),
                Err(_) => { println!("Error !") }
            }
            if let Err(_) = i{
                println!("Error !");
                return Ok(());
            }
            let i = i.unwrap();
            total_send += i;
            if total_send >= size {
                break
            }
        }
        println!("{} bytes send !", total_send);
        // stream.flush()?; // Doesn't seem to work
        stream.shutdown(std::net::Shutdown::Both)?;
        if delay {
            thread::sleep(short_time);
        }
        println!("Go !");
    }

    Ok(())
}

Thank you.

[alice]

It's because you are not reading the request sent to you. Read the entire request, then send the data. That will work.

Your loop is more complicated that it has to be, you can do this instead:

use std::cmp::min;

while total_sent < size {
    let ps = min(packet_size, size - total_sent);
    total_sent += send_file(&mut file, &mut stream, ps)?;
}

[ache]

I realise that the best I can do was to wait for the client to send EOF, optionaly with a timeout. I also think that reading the request is better than what I was trying to do.

I tried to read the request with stream.read_to_end(&mut req); but it's blocking. Wainting for EOF I guess.
I will read other implementations of HTTP server (tinyhttp, hyper) to see if they use a buffer, read bytes by bytes or something else to handle the request.

Thank you for the tip about the while loop !

Will post as soon as I get something that works ^^

[alice]

Keep reading the stream until you see a double newline. Then you know the request is done.

[ZiCog]

An HTTP request contains a length field in the headers. For example:

Content-Length: 1234

That length is the actual length of the data put in the message body.

The body is anything sent after the blank line and the end of the headers. See HTTP Messages - HTTP | MDN

As far as I can tell the length header is not required if there is no body.

You should parse the request headers and read that exact length of bytes from the body then close the connection.

If there is no header filed close the connection after detecting the blank line at the end of the headers.

[ache]

@alice

It's because you are not reading the request sent to you. Read the entire request, then send the data. That will work.

O_o It appears to be exactly true.
However, even reading only the first line of the request header solves my problem.
I don't even need to wait for the client to close the connection. I just need to read the request. I don't understand why. Maybe it takes a few millisecond and that the time needed to send everything or maybe something else.

@alice

Keep reading the stream until you see a double newline. Then you know the request is done.

I assume that by double newline, you are talking about CRLF. Then you are right. It's what tinyhttp does (i checked and it's written in the RFC 2068).

@ZiCog: ? A GET request should not have a body part. In fact it may have one but that's a non-sense.
Didn't really understand your point

My goal here is not to implement a HTTP server but to benchmark the sendfile syscall. I understand that I shouldn't care a lot about closing the TcpStream but I still doesn't understand that bug. For information my code is now :

use std::{io, ptr};
use std::os::unix::io::AsRawFd;
use std::fs::File;
use std::net::TcpListener;
use std::convert::TryInto;
use std::cmp::min;
use std::io::{BufReader, BufRead, Write};
use std::time::Instant;


pub fn send_file<R: AsRawFd, W: AsRawFd>(r: &mut R, w: &mut W, size: usize) -> io::Result<usize> {
    match unsafe { libc::sendfile(w.as_raw_fd(), r.as_raw_fd(), ptr::null_mut(), size) } {
        -1 => Err(io::Error::last_os_error()),
        copied => Ok(copied as usize),
    }
}

fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:8000")?;
    println!("Connected on localhost port 8000");

    // accept connections and process them serially
    for stream in listener.incoming() {
        let start = Instant::now();
        let mut stream = stream.unwrap();
        let mut req_header = String::new();

        let mut reader = BufReader::new(&stream);

        loop {  // I should use `while` using the number of bytes readed by read_line
            let mut tmp_buf = String::new();
            reader.read_line(&mut tmp_buf)?;
            println!("({}) [{}]", tmp_buf, tmp_buf.len());
            if tmp_buf == "\r\n" {
                break
            } else if tmp_buf == "" { // To stop if the connection is closed
                return Ok(())
            } else {
                req_header += &tmp_buf;
            }
        }

        println!("Req : {}", req_header);

        let http_header_first = "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\nContent-length: ";
        let http_header_last = "\r\nServer: testSRV/0.1\r\nAccept-Ranges: bytes\r\n\r\n";

        let mut file = File::open("big")?;
        let metadata = file.metadata()?;
        let size : usize = metadata.len().try_into().unwrap();
        let http_header = http_header_first.to_owned() + &size.to_string() + http_header_last;

        std::println!("File of {} bytes", size);
        stream.write(http_header.as_bytes())?;
        let mut total_sent = 0;
        let packet_size : usize = 2<<15;

        println!("Packet size: {}", packet_size);

        while total_sent < size {
            let ps = min(packet_size, size - total_sent);

            total_sent += send_file(&mut file, &mut stream, ps)?;
        }

        println!("{} bytes send !", total_sent);

        let d = start.elapsed();
        println!("Done ! {:?}", d);
    }

    Ok(())
}

Thank you for you help ! I'm happy that I can now serve static files better than a Go HTTP server.