Define a function with generics and lifetimes (HRTB issues)

help
1

Hi everyone,
I am trying to manually implement a packet deserializer for a TCP stream using a partial zero copy approach. For this I decided to have a struct for each packet which will contain references to a buffer in memory.
I have also defined a deserialize trait to implement in every packet struct.

pub struct TCPClientAnnounce<'a> {
    service_name: Cow<'a, str>,
}

pub trait DeserializeBinary<'a> : Sized {
    fn deserialize(data: &'a [u8]) -> Result<(Self, usize), Error>;
}

impl<'a> DeserializeBinary<'a> for TCPClientAnnounce<'a> {
    fn deserialize(data: &'a [u8]) -> Result<(Self, usize), Error> {
        ...
    }

Since I have multiple packets, I also decided to implement a helper function that would execute a closure once all bytes have arrived:

async fn handle_next_packet<'a, T, F>(buffer: &'a mut Vec<u8>, tcp_stream: &mut TcpStream, mut handler: F) -> Result<(), GCLogNetworkError>
where
    T: for<'de> DeserializeBinary<'de>,
    F: FnMut(T),
{
    loop {
        tcp_stream.read(buffer).await?;
        let result = T::deserialize(buffer);
        match result {
            Ok((packet, size)) => {
                handler(packet);
                buffer.drain(..size);
                return Ok(());
            }
            // Non related error handling
        }
    }
}

Up until now everything seems to compile perfectly, but once I try to use this new function:

let mut buf = vec![0; BUFFER_PER_CONNECTION];

let service_name: String = String::new();
handle_next_packet(&mut buf, &mut socket, |p: &TCPClientAnnounce|{
    service_name = p.service_name().to_string();
}).await?;

I get the following error:

implementation of `DeserializeBinary` is not general enough
`DeserializeBinary<'0>` would have to be implemented for the type `TCPClientAnnounce<'_>`, for any lifetime `'0`...
...but `DeserializeBinary<'1>` is actually implemented for the type `TCPClientAnnounce<'1>`, for some specific lifetime `'1`

Here I understand the error (I think), the lifetime of T should be specific to 'a. But if I remove the the HRTB and set the lifetime of T to be 'a, I get a lifetime error:

async fn handle_next_packet<'a, T, F>(buffer: &'a mut Vec<u8>, tcp_stream: &mut TcpStream, mut handler: F) -> Result<(), GCLogNetworkError>
where
    T: DeserializeBinary<'a>,
    F: FnMut(T),
{
    loop {
        tcp_stream.read(buffer).await?; // lifetime error: cannot borrow `buffer` as mutable more than once at a time
        match result {
            Ok((packet, size)) => {
                handler(packet);
                buffer.drain(..size);
                return Ok(());
            }
            // Non related error handling
        }
    }
}

Is there a way I can make this work without resourcing to unsafe or owning data? This is not the first time that I encounter a similar problem, I still may have something to learn about HRTB's.
Also, I understand that are crates to this kind of stuff, but I plan to do this without any dependency and learn trough the process.

2

The problem with the first version is that T can't be a single type that outlives the function call; it has to vary depending on what lifetime it was constructed with.

The problem with the second version is that 'a can't be a single lifetime that outlives the function call; it has to be different for each loop iteration.

In general, you have to write your function in a way in which 'de is not used in any of the function’s generic parameters, because by definition, a function’s generic parameters all outlive the function call, and have a single value over the whole function call.

There isn’t a really clean solution to this problem. You can use a GAT trait to describe the family of types you need, using a placeholder type to make it identifiable:

trait DeserializeHelper {
    type Value<'a>: DeserializeBinary<'a>;
}

// this self type is a placeholder that is never actually created,
// so the 'static doesn't matter
impl DeserializeHelper for TCPClientAnnounce<'static> { 
    type Value<'a> = TCPClientAnnounce<'a>;
}

async fn handle_next_packet<'a, T, F>(buffer: &'a mut Vec<u8>, tcp_stream: &mut TcpStream, mut handler: F) -> Result<(), GCLogNetworkError>
where
    T: DeserializeHelper,
    F: for<'de> FnMut(T::Value<'de>),
{
2 Likes
3

Thanks a lot for the explanation. Seems weird that there isn't a mechanism to solve this problem more easily.

Also, in your snippet I guess you made a typo and should be this:

// Instead of impl DeserializeBinary for TCPClientAnnounce<'static>
impl DeserializeHelper for TCPClientAnnounce<'static>

Or am I missing something?

4

Oh, yes, sorry, you have identified the error correctly.

1 Like