Read struct from bytes

I'm writing an editor program and need to read a series of structs from a file (in big endian), for example:

struct Header {
    file_magic: [u8; 4],
    file_len: u32,
    num_sections: u16,
    header_len: u16,
    version_num: u32,
    section_offsets: [u32; 15],
}

I initially manually created functions to read these from the file, but thought there must be a way of writing a generic function to do this. My current approach uses a macro like this:

macro_rules! read_struct {
    ($r:expr, $t:ty, $msg:expr) => {{
        let mut encoded = [0u8; mem::size_of::<$t>()];
        $r.read_exact(&mut encoded)?;
        let result: $t = bincode::DefaultOptions::new()
            .with_fixint_encoding()
            .with_big_endian()
            .allow_trailing_bytes()
            .deserialize(&encoded)
            .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, $msg))?;
        result
    }};
}

I had to use a macro because I couldn't find another way of creating an array with the size of the generic type. I couldn't use bincode's deserialize_from function because I need to only read the exact number of bytes from the reader to fill the struct, and no more than that.
Surely there's a better way of doing this though?

bincode is its own format, which is different from layout and size of Rust types. mem::size_of and number of bytes read by bincode are unrelated.

IIRC bincode only reads as many bytes as it needs, so you can stream in bincode types. You just won't get to use read_exact, and bincode will issue many read operations.

You can read any generic struct using bincode by using serde's DeserializeOwned trait bound, or bincode's own Decode trait.

Alternatively, if you want to read something represented by the Rust struct, you must use #[repr(C)] on the struct, as otherwise the actual order of fields in a struct can be different from the order in the source code (it can even be literally random!). See MaybeUninit type for allocating a space with the right size and alignment for a Rust struct.

2 Likes

Thanks, can you give an example of a read_struct function that would fix the issues with my approach? Sorry I'm quite new to this!

You might be interested in the zerocopy crate and its byteorder module/feature zerocopy::byteorder - Rust

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.