Using tokio to mix reading lines and binary

[blonk]

I'm writing an application that sends and receives data mixed line-based and binary data. In short, the protocol looks like this:

Command\n
Key1 Value1\n
...
KeyN ValueN\n
\n

Some of them may indicate that binary data follows:

HereIsAFile\n
Len 65536\n
\n
<64K of binary data>

Several of these may be sent consecutively and the data may be arbitrarily long.

What I'm wondering is how one would go about switching between the lines-based format and the binary. I had imagined that I would use the following construction:

let (recv, send) = io::split(stream);
let reader = io::BufReader::new(recv);
let mut lines = reader.lines();
if let Some(line) = lines.next_line().await? {
  // populate header until an empty line is encountered
}

When I need to switch over from "lines mode" to binary mode I would call lines.into_inner() to get the BufReader back, but then I saw this:

Unwraps this Lines<R> , returning the underlying reader.

Note that any leftover data in the internal buffer is lost. Therefore, a following read from the underlying reader may lead to data loss.

I'm unsure how to parse that last bit, but it sounds like the line reader consumes more data than it uses to return lines (if one switches before getting all the lines in the buffer).

Or is it simply saying "Just after you've called into_inner() you should extract whatever data you want from the buffer before calling read again?

Should I avoid switching between the two modes and use some other means?

[blonk]

Turns out this was more complicated than I thought.

Also seems that the line reader in Tokio makes the assumption that it if it encounters one line ending, it blindly assumes that it can take away the next one as well if it means it's a CRLF, which means an arbitrary binary file can't be sent just after the line parser.

Back to the drawing board.

[alice]

I would not use Lines for this purpose. The codec module in tokio-util is a better bet.