Cannot implement trait on Vec<u8> and fixed sized array [u8; N]

I implemented BinaryConverter trait that provides me methods to read from/into buffer:

pub trait BinaryConverter {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error>;
    fn read_from<R: BufRead>(&mut self, reader: R) -> Result<Self, Error> where Self: Sized;
}

Next I implemented it on each type I want to use with. But the issue there is that impl not work for Vec and I got next error on compile:

error[E0277]: the trait bound `Vec<u8>: BufRead` is not satisfied
   --> src/main.rs:183:55
    |
183 |                           $field_name: self.$field_name.read_from(&mut buffer).unwrap(),
    |                                                         ^^^^^^^^^ the trait `BufRead` is not implemented for `Vec<u8>`
...
193 | /     packet! {
194 | |         properties {
195 | |             opcode 100;
196 | |             size: u16;
...   |
205 | |         }
206 | |     }
    | |_____- in this macro invocation
    |
    = help: the following other types implement trait `BufRead`:
              &[u8]
              &mut B
              Box<B>
              BufReader<R>
              StdinLock<'_>
              std::io::Chain<T, U>
              std::io::Cursor<T>
              std::io::Empty
              std::io::Take<T>
    = note: required because of the requirements on the impl of `BufRead` for `&mut Vec<u8>`
note: required by a bound in `BinaryConverter::read_from`
   --> src/main.rs:6:21
    |
6   |     fn read_from<R: BufRead>(&mut self, reader: R) -> Result<Self, Error> where Self: Sized;
    |                     ^^^^^^^ required by this bound in `BinaryConverter::read_from`
    = note: this error originates in the macro `packet` (in Nightly builds, run with -Z macro-backtrace for more info)

First of all, I want to mention, I did impl on Vec<u8> instead of fixed sized array, because I am not sure how to do this. So each time I faced with smth like [u8; 10] I do .to_vec() call on it.

But, anyway, I still need both: Vec and fixed sized array.

Now, this is full example of my trait:

pub trait BinaryConverter {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error>;
    fn read_from<R: BufRead>(&mut self, reader: R) -> Result<Self, Error> where Self: Sized;
}

impl BinaryConverter for u8 {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_u8(*self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<u8, Error> {
        reader.read_u8()
    }
}

impl BinaryConverter for u16 {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_u16::<LittleEndian>(*self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<u16, Error> {
        reader.read_u16::<LittleEndian>()
    }
}

impl BinaryConverter for u32 {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_u32::<LittleEndian>(*self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<u32, Error> {
        reader.read_u32::<LittleEndian>()
    }
}

impl BinaryConverter for u64 {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_u64::<LittleEndian>(*self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<u64, Error> {
        reader.read_u64::<LittleEndian>()
    }
}

impl BinaryConverter for String {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_all(&self.to_string().into_bytes())
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<String, Error> {
        let mut internal_buf = vec![];
        reader.read_until(0, &mut internal_buf)?;
        match String::from_utf8(internal_buf) {
            Ok(string) => Ok(string),
            Err(err) => Err(Error::new(ErrorKind::Other, err.to_string())),
        }
    }
}

impl BinaryConverter for Vec<u8> {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_all(self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<Vec<u8>, Error> {
        let mut internal_buf = vec![];
        reader.read_exact(&mut internal_buf)?;
        Ok(internal_buf)
    }
}

I use it in my macro, that generates structs like below:

packet! {
	properties {
		opcode 100;
		size: u16;
	}
	
	#[derive(Hash)]
	pub struct Outcome {
		name: String,
		os: u8,
		version: u32,
		game_name: String,
	}
}

let mut packet = Outcome {
	name: String::from("Some name"),
	os: 1,
	version: 9,
	game_name: String::from("Game"),
};

This is sandbox.

This is my macro:

macro_rules! packet {
    // first rule
    (
        properties {
            $(opcode $opcode_value:expr;)?
        }
        
        $(#[$outer:meta])*
        $vis:vis struct $PacketStruct:ident {
            $($field_name:ident: $field_type:ty),*$(,)?
        }
        
        $($StructImpl: item)*
    ) => {
        $(#[$outer])*
        #[derive(Clone)]
        $vis struct $PacketStruct {
            $($field_name: $field_type),*
        }
        
        $($StructImpl)*
        
        $(
            impl $PacketStruct {
                pub fn get_opcode() -> u32 {
                    $opcode_value
                }
            }
        )?
        
        impl $PacketStruct {
            pub fn to_binary(&mut self) -> Vec<u8> {
                let mut packet = Vec::new();
                $(
                    self.$field_name.read_into(&mut packet);
                )*;
                packet
            }
        }
    };
    // eof first rule
    // second rule
    (
        properties {
            opcode $opcode_value:expr;
            $($prop_name:ident: $prop_type:ty;)*
        }
        
        $(#[$outer:meta])*
        $vis:vis struct $PacketStruct:ident {
            $($field_name:ident: $field_type:ty),*$(,)?
        }
        
        $($StructImpl: item)*
    ) => {
        packet! {
            properties {
                $($prop_name: $prop_type;)*
            }
            $(#[$outer])*
            $vis struct $PacketStruct {
                $($field_name: $field_type),*
            }
            
            $($StructImpl)*
            
            impl $PacketStruct {
                pub fn get_opcode() -> u32 {
                    $opcode_value
                }
            }
        }
    };
    // eof of second rule
    // third rule
    (
        properties {
            $($prop_name:ident: $prop_type:ty;)*
        }
        
        $(#[$outer:meta])*
        $vis:vis struct $PacketStruct:ident {
            $($field_name:ident: $field_type:ty),*$(,)?
        }
        
        $($StructImpl: item)*
    ) => {
        $(#[$outer])*
        #[derive(Clone)]
        $vis struct $PacketStruct {
            $($field_name: $field_type),*
        }
        
        $($StructImpl)*
        
        impl $PacketStruct {
            pub fn to_binary(&mut self) -> Vec<u8> {
                let mut packet = Vec::new();
                $(
                    self.$field_name.read_into(&mut packet);
                )*
                packet
            }
            
            pub fn from_binary(&mut self, buffer: Vec<u8>) -> $PacketStruct {
                $PacketStruct {
                    $(
                        $field_name: self.$field_name.read_from(&mut buffer).unwrap(),
                    )*
                }
            }
        }
    };
}

Could somebody explain, how to fix the issue and how to implement this trait on fixed sized array ?

Just look at the error message and what it points to:

    |
183 |                           $field_name: self.$field_name.read_from(&mut buffer).unwrap(),
    |                                                         ^^^^^^^^^ the trait `BufRead` is not implemented for `Vec<u8>`
...

The problem isn't that you didn't implement the BinaryConverter trait for Vec<u8> – you clearly did. The problem is that read_from() takes an argument that must implement BufRead, because you said so. However, Vec<u8> doesn't. A possible workaround is to wrap it in an io::Cursor, which does:

pub fn from_binary(&mut self, buffer: Vec<u8>) -> $PacketStruct {
    let mut cursor = Cursor::new(buffer);
    $PacketStruct {
        $(
            $field_name: self.$field_name.read_from(&mut cursor).unwrap(),
        )*
    }
}

By the way, there are several other things in your code which are wrong or at least weird and/or non-idiomatic:

  • the BinaryConverter::read_into() function takes &mut self whereas it would be enough to take &self – you are not modifying the value, you are only writing it to a buffer. Relatedly, the name of this function should be write_into(), not read_into().
  • the BinaryConverter::read_from() function also takes a self parameter, but that's nonsense – how are you supposed to already have instances of the implementing type if what you are trying to do is exactly to construct it?
  • the implementation for String is just completely wrong. You are writing out the string contents without any indicator of its length or a terminator, but when reading it back, you are using reader.read_until(0, &mut internal_buf). This means that you are implicitly assuming that strings are 0-terminated, which they aren't. What's more, using 0 as the sentinel doesn't work, because unlike C strings, Rust strings can contain embedded zeros.
  • relatedly, the read_into() function implemented on String copies the string (and thus allocates) unnecessarily. Why don't you implement it as buffer.write_all(self.as_bytes())?
  • the implementation for Vec<u8> is also wrong. The read_exact() method of readers tries to fill the buffer with as many bytes as its length. You are always passing in an empty buffer, so you will never actually read anything.
  • these last 3 points suggest that you didn't think about how to represent the length of variable-length types at all – you should most likely do that, preferably by prepending the length of the value to its binary representation.
  • the inherent from_binary() method also unnecessarily takes a self argument, and it's also unnecessary for it to accept an owned Vec<u8>. You could just take a slice instead.
  • similarly, the inherent to_binary() method doesn't need to take &mut self. It should just take &self.

By the way, are you sure you want to implement this all by yourself instead of using Serde with an appropriate binary format such a Bincode or MessagePack?

You do it in the exact same way as you would do it with any other type:

impl<const N: usize> BinaryConverter for [u8; N] {
    fn read_into(&mut self, buffer: &mut Vec<u8>) -> Result<(), Error> {
        buffer.write_all(self)
    }

    fn read_from<R: BufRead>(&mut self, mut reader: R) -> Result<Self, Error> {
        let mut internal_buf = [0; N];
        reader.read_exact(&mut internal_buf)?;
        Ok(internal_buf)
    }
}

By the way, you should really not spell out the concrete return type in the implementation of read_from(); you should just stick to the trait's signature of returning -> Result<Self, Error>, because:

  • this makes it more consistent with the trait signature
  • it will be clearer to the uninitiated reader that the purpose of this method is to construct a value of the type from a byte stream. You don't always construct "a string" or "a u32" in this method – you always construct the type it is implemented on.
  • it will be less annoying to implement the trait for new types by copy-pasting, because there will be one fewer place in the code where you have to change the type.
1 Like

Thank you for detailed explanation !

the issue with Serde + Bincode is that sometimes I need to deserialize fields with dynamic size.
For example, I need to deserialize such packet:

let mut server_ephemeral = vec![0u8; 32];
reader.read_exact(&mut server_ephemeral)?;
let g_len = reader.read_u8()?;
// this field size depends on g_len value
let mut g: Vec<u8> = vec![0u8; g_len as usize];
reader.read_exact(&mut g)?;
let n_len = reader.read_u8()?;
// this field size depends on n_len value
let mut n: Vec<u8> = vec![0u8; n_len as usize];
reader.read_exact(&mut n)?;
let mut salt = vec![0u8; 32];
reader.read_exact(&mut salt)?;

or packet where I have a string at the beginning:

let id = reader.read_u64::<LittleEndian>()?;
let mut name = Vec::new();

reader.read_until(0, &mut name)?;

let race = reader.read_u8()?;
let class = reader.read_u8()?;
let gender = reader.read_u8()?;

I am not sure how to solve such issues using serde + bincode. The only case where bincode could help it's when dynamic sized field is at the end of fields sequence, so I can do smth like described here.

BTW, I tried already to ask if this possible here.

regarding null-terminated strings. All string I read from/write to server are null-terminated, it's how the server implemented (server written in C++). My project is TCP client that just interacts with this server. So I am sure at least most of the strings are null-terminated.

There's nothing in particular to be "solved" here. Bincode supports data with dynamic length such as String and Vec.

1 Like

Well, seems like macro is not need for what I want to implement. But anyway, it was good experience because one of my goals was to learn how to implement macroses in Rust.

I just tried serde + bincode on some structs and seems like this crates evaluate proper amount of all fields (including null-terminated strings and fixed sized array).

I just found, bincode serealizing string in another way than I need. It puts u64 value equals to string length together with encoded string. Do you know does it possible to configure String serealizing (I need to store null-terminated string instead) ?

No, that'd be another format, then.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.