Is there any resources for tcp byte framing with actual delimiters (not length delimited)?
I am trying to make a file transfer system and i need a way to properly frame it.
The resources I have found so far were for length delimiting.
E.g //f and \f
it's relatively simple to do, first of all you choose a delimiter, then you need a rule to escape it in case it's present in the data you want to send and finally the reciever needs to have a state machine that keeps reading until it sees the delimiter at witch point you have your data and just have to reverse the escaping something like this
let next_frame:Vec<u8>=Vec::new();
let mut broken_frame=true;//in case the stream ends too early
while let Some(next_byte) =tcp_stream.next().await
{
if next_byte==delimiter_byte{
broken_frame=false;
break
}
next_frame.push(next_byte);
}
if broken_frame{
return Err(());
}
invert_escape_process(next_frame);
Ok(next_frame)
i wrote it down a bit fast and loose but hopefully should be enough to convey the general concept
This is because the underlying networking APIs are more difficult and/or inefficient to use with character-based delimiters.
If you know how many bytes to read, you can allocate a buffer and fill it in one syscall. If you need to scan, you either have to ask for one byte at a time (thousands times slower) or implement support for partial reads and scans of temporary buffers.
Length-delimited framing works even if you're streaming data of unknown length - you can have variable-length chunks with an empty chunk as a terminator.
Usual XY comment: Why do you want to do this?
If you're inventing your own protocol anyway, consider just not using TCP if you need multiple parts. For example, you could send each file in a separate QUIC stream and not need to delimit it yourself in the first place.
Or you generate it so that it's probabilistically impossible for it to be in the data https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html.
HTTP multipart, I'd believe.
The HTTP multipart that is still in use is sent inside a length-delimited body, which often is sent inside length-delimited frames.
A terminator that is statistically practically impossible to occur in any data stream would need to be longer than 4-8 bytes you need for a length.
Even if you're sending 1-byte messages, a variable-length integer will be as efficient as a minimal byte terminator, without all the downsides.
You don't necessarily have to choose between having a known length ahead of time and length-encoded framing; http chunked transfer encoding is an easy to understand example of sending an unknown amount of data without needing fiddly escaping methods.
Wikipedia's example is a bit hard to read, but formatted it's just "Wikipedia in \r\nchunks." being sent as:
4
Wiki
7
pedia i
B
n
chunks.
0
(with CRLF line endings)
You could use existing binary level framing for the same effect.
I use a very simple approach - 4 bytes packet deimitter. A packet boundary is PACKET_END. Another boundary is for canceling connection, and you can ignore it for your case. Generally, I could introduce a sub protocol on top of the standard websocket protocol, but since it has scope - one application, there is no reason doing so. I guess you can introduce own delimiters and way to handle the target packet. My packet is JSON, but any other packet payload will work too.