Tokio and protocols vs. traditional stream parsing


So I’m getting a pretty good grip on Rust lately, I think. A lot of the stuff I made formerly in C/C++ were usually servers or clients for a type of network. When I was a novice in this, I started out with naive attempts using Qt’s QDataStreams and parsing binary data from a QTcpSocket, while hooking up events with it’s signals/slots feature. Eventually, I realised the relevence of asynchronicity when sending data and, also yearning for a more flexible license and less DLL/dependency-heavy, switched to Boost’s ASIO library, buffering my data with strands and worker threads, etc.

Now that I’m on Rust, I’m seeing the top contenders for similar functionality as QtNetwork/Boost.ASIO would be Tokio and MIO. MIO flat out states it is low-level and refers me to Tokio for more easier functionality. So I go to look at Tokio, which seems to have a pretty straightforward example on its front page. However, the tutorial says tokio_proto is recommended for newbies. I looked over how codecs and protocols are coded, and while it seems… understandable… I’m rather concerned about how this codec/protocol model sends and receives data. It seems to send/receive all data in one structured format (ie. all data must be a line of UTF8 text) as opposed to, perhaps, an initial ‘hello’-type message that contains metadata and THEN the string messages going back and forth.

I think a simple example of what would help me understand it better is… how would one go about making a minimalistic Hotline client? The protocol spec (found here) basically starts with the client sending the bytes ‘T’, ‘R’, ‘T’, ‘P’, 0, 1, 0, 2 as a greeting, with the server replying with ‘T’, ‘R’, ‘T’, ‘P’ if successful, and THEN it goes into an event loop where the data is structured. How would tokio_proto go about doing that as well as heeding the greeting data? Because, as you can see, the data isn’t always sent in a fixed format.

I suppose, perhaps, instead, it would be better to find out how Tokio can simply send and receive buffered data asynchronously instead of all the codec/protocol gobbledeegook.


I must admit I have not used tokio-proto at all and have only been working directly with tokio-io constructs, but I can at least talk about how to do a hello message and switch to a structured Codec.

The simplest way to do something like this is to have a multi-phase client, I have something sort of similar (in a private repo still because I don’t want to make crypto related stuff public without giving it a more thorough review) where I’m handshaking an encrypted stream. The initial handshake is using one form of structured messages, then once the encryption parameters have been exchanged the stream switches over to a different form of structured messages. A greatly simplified example:

struct HandshakeCodec // : Encoder<Item=Vec<u8>> + Decoder<Item=Vec<u8>>
struct StreamCodec    // : Encoder<Item=Vec<u8>> + Decoder<Item=Vec<u8>>

struct SecStream<S: AsyncRead + AsyncWrite + 'static> {
    inner: Framed<S, StreamCodec>,
    parameters: Vec<u8>,

impl<S: AsyncRead + AsyncWrite + 'static> SecStream<S> {
    fn new(parameters: Vec<u8>, transport: FramedParts<S>) -> SecStream<S> {
        let inner = Framed::from_parts(transport, StreamCodec);
        SecStream { inner, parameters }

pub fn handshake<S>(transport: FramedParts<S>, my_parameters: Vec<u8>)
    -> impl Future<Item=SecStream, Error=io::Error> 
    where S: AsyncRead + AsyncWrite + 'static
    Framed::from_parts(transport, HandshakeCodec)
        .and_then(|transport| transport.into_future())
        .map(|(their_parameters, transport)| SecStream::new(their_parameters, transport.into_parts()))

SecStream then implements Stream + Sink via encrypting/decrypting messages using the parameters and passing them through to the inner framed stream. I believe this could then be used as part of a {Server/Client}Proto, but as I said I never looked into tokio_proto as it doesn’t really fit my usecase.

(This example also takes in a FramedParts because the stream it uses previously had another form of structured messages used while negotiating which encryption to use, this is something that I hope becomes more implicit in the future, and may causes issues with tokio_proto passing the stream in).


This is what you are looking for. Basically, you have read the echo example, which uses a non-streaming pipelined protocol. Yet if you choose a streamed, multiplexed protocol, you can have a ‘greeting’ message frame, and then have an event loop where the body is structured in chunks.