Implementing a communication protocol

I want to implement a communication protocol we use at work. There are already implementations in C and Python. I came acros this post, describing the same task.

I can roughly imagine the idea but am not sure how to proceed.

Protocol

  • Every packet has a common header (1 to 3 bytes long)
  • The packets body is different, based on its type (set in the header)
  • there are around 9 types of service packets
  • Packets can also be empty

I thought about something like this:

enum ServicePacket {
    Unknown(BasePacket),
    Network(NetworkPacket),
    ...
}

struct BasePacket;
struct NetworkPacket;

But then where would I store the headers? In an OOP language it could be an inherited attribute...
To force common parsing etc. I tought about using:

trait Packet {
    fn from_bytes(bytes: &[u8]) -> Result<Self, Error>
    where 
        Self: Sized;
}

impl Packet  for BasePacket {} and so on...

Is this something I could extend on or am I into a dead end?
In the end I want to have a library that I can use to build a node in that protocols network and communicate with the others.
Any hint, advice and idea is very welcome.

But then where would I store the headers?

Every packet has this header? Then how about wrapping the enum?

struct Packet {
  header: PacketHeader,
  payload: ServicePacket,
}

Is this a known/public protocol? Maybe there already are some implementations that you could use or be inspired by?

To force common parsing etc.

Somewhere you'll have to make a case distinction on the type of the packet that you're receiving to be able to invoke the correct parser.

No, it is in internal protocol but there are Python and C implementation.

True, that would work.
Can I specify, that the payload has to be something that implements e.g. the trait ServicePacket and then I could just implement that trait for every ServicePacket. Like parsing and format...

struct Packet<T: ServicePacket> {
    header: PacketHeader,
    payload: T,
}

trait ServicePacket {
    fn parse(bytes: &[u8]);
}

Then I would just define a struct for every ServiceType?

struct NetworkPacket {
    let some_fields: u8,
}

impl ServicePacket for NetworkPacket {
     ....
}

But I feel like there has to be another, more....ergonomic way with enums, since I know all possible variants of the services.

Anonymised from an in-house protocol:

struct Packet {
    header: PacketHeader,
    payload: PacketPayload
}

enum PacketPayload {
    DummyPacket,
    ConfigPacket { … },
    OtherPacketType(InteriorData),
}

That seems nice.
I saw the python implementation do it the other way around.

struct BasePacket {
    header: Header,
    raw_payload: [u8],
}

struct SomeServicePacket {
    base_packet: BasePacket
    spcecific_fields: dtype
    ...
}

impl SomeServicePacket {
    pub fn from_base_packet(&mut self, packet: BasePacket) {
        ...
    } 
}

But that would deny the possibility to pattern match the different packets right?
So the combination could be:

enum ServicePacket {
    Unknown,
    Network(NetworkPayload),
}

struct BasePacket {
    service_id: u8,
}

impl BasePacket {
    pub fn into_service(mut self) -> Result<ServicePacket, std::io::Error> {
        match self.service_id {
            0 => Ok(ServicePacket::Network { 0: NetworkPayload }),
            _ => Ok(ServicePacket::Unknown),
        }
    }
}

struct NetworkPayload;

fn handle_packet(packet: BasePacket) {
    if let Ok(service_packet) = packet.into_service() {
        match service_packet {
            ServicePacket::Unknown => todo!(),
            ServicePacket::Network(_) => todo!(),
        }
    }
}

Idea 1

Extending this I would just have to implement the payload fields for every distinctive packet type and their respective parsing in the into_service. Which could itself be done by the payload struct themself when they implement a common trait Payload { from_raw_bytes() }

trait Payload {
    fn from_bytes(bytes: &[u8]) -> Self;
}

impl Payload for NetworkPayload {
    fn from_base_packet(bytes: &[u8]) -> { todo!() }
}

impl BasePacket {
    pub fn into_service(mut self) -> Result<ServicePacket, std::io::Error> {
        match self.service_id {
            0 => Ok(ServicePacket::Network { 0: NetworkPayload::from_base_packet(self)}),
            _ => Ok(ServicePacket::Unknown),
        }
    }
}

Idea 2

Also the BasePacket could implement .into() for every ServicePacket.

impl Into<NetworkPayload> for BasePacket {
    fn into(self) -> NetworkPayload {
        todo!()
    }
}

impl BasePacket {
    pub fn into_service(mut self) -> Result<ServicePacket, std::io::Error> {
        match &self.packet_type {
            0 => Ok(ServicePacket::Network { 0: self.into() }),
            _ => Ok(ServicePacket::Unknown),
        }
    }
}

I am just playing with some ideas that came into my mind while writing this. Is there any idomatic way or should I just go ahead and see where it leads me.

IME, there's two good routes, depending on whether you're receiving a byte stream, or a packetised stream.

For byte streams, the way tokio_util::codec does things is great; you have an outer layer which buffers bytes from the stream, and presents the buffer to the decoder, which can output either None to indicate more bytes are needed to decode a full packet, or Some(packet) to return a fully decoded packet to the next layer out. You implement a Decoder, and have a different layer do the I/O; this also means that you can test the decoder without doing I/O.

For packetised streams, each packet should decode to exactly one payoad. You'd therefore implement a fn decode_payload(bytes: &[u8]) -> Result<ServicePacket, Error>, which decodes a block of bytes taken from the packetized stream into a ServicePacket, or errors if you can't do this.

In either case, your packet data structure is something like:

enum ServicePacket {
    Unknown { service_id: u8 },
    Network(NetworkPayload),
}

impl ServicePacket {
    fn service_id(&self) -> u8 {
        match self {
            Self::Unknown { service_id } => service_id,
            Self::NetworkPayload(_) => NETWORK_SERVICE_ID,
    }
}

(assuming that the header contents can be derived from the payload - e.g. if it's just service_id and size), or if the header has other interesting fields:

struct ServicePacket {
    header: Header,
    payload: ServicePayload
}

enum ServicePayload {
    Unknown(Vec<u8>),
    Network(NetworkPayload),
    …
}

impl std::convert::AsRef<ServicePayload> for ServicePacket {
    fn as_ref(&self) -> &ServicePayload {
        &self.payload
    }
}

(noting that you might want an inherent method, rather than an AsRef implementation, I'm just showing an AsRef implementation because I can). This lets you extract the payload to examine via match and similar, while still keeping the header in a common struct.

1 Like

The only thing I know is, that the packets should be parsed from different sources lik TCP, Bluethooth, Serial, i2c and so on. The packets themself are within the payload of the respective network protocols.

The headers are definitely important and should be available in the service packets.

I think I will go forward with your approach, seems to fit the usecase for now.

Given that you might be getting raw byte streams (serial, TCP), I'd suggest the Decoder route as your core, with adapters to the various byte streams, and to decode from complete packets.

1 Like

So I can build a TCPDecoder that turns the bytes from my tcp connection into the custom protocol packets and has the type Item = ServicePacket. Then it does not matter from which network protocol I receive the packet. Neat

1 Like