How to load dynamic chunk sized binary file

[vahvero]

I am attempting to read a binary file with multiple possible integer sizes. I cannot seem to find a way to do this elegantly.

It would be something like this, but obviously this does not work.

fn read_data<T>(data: &Vec<u8>) -> Vec<T> {
        let mut data = Vec::new();
        let _ = file.read_to_end(&mut data);
        data.chunks(T::my_byte_size)
            .map(|x| {
                let mut a = [0; T::my_byte_size];

                for i in 0.. T::my_byte_size {
                    a[i] = x[i];
                }
                T::from_le_bytes(a)
            })
            .collect()
}

Forcing the T size would however

fn read_data(data: &Vec<u8>) -> Vec<i64> {
        let mut data = Vec::new();
        let _ = file.read_to_end(&mut data);
        data.chunks(8)
            .map(|x| {
                let mut a = [0; 8];

                for i in 0..8 {
                    a[i] = x[i];
                }

                i64::from_le_bytes(a)
            })
            .collect()
}

How should this be implemented in rust?

[jameseb7]

You can implement your first approach using the FromBytes trait from the num_traits crate. You need some way to produce the FromBytes::Bytes type though. An easy way is to require Zero and ToBytes, where the Bytes types are the same, then convert to &[u8] for reading.

Something like this:

fn read_data<T>(mut file: &[u8]) -> Result<T, Error>
where
    T: FromBytes,
    <T as FromBytes>::Bytes: Sized,
    T: ToBytes<Bytes = <T as FromBytes>::Bytes> + Zero,
{
    let mut data = T::zero().to_le_bytes();
    file.read_exact(data.as_mut())?;
    Ok(T::from_le_bytes(&data))
}

(playground)

Note that this just reads a single value, rather than a vector converting the whole file to a particular type, but it should be adaptable or usable in a loop. Also, note that the return value of read_exact() is propagated to the caller rather than ignored (it will fail if file isn't long enough for the desired type). I also changed the input type to be more general (a reference to a Vec doesn't give you more than &[u8], but prevents passing references to other array types).

[jumpnbrownweasel]

@vahvero

It would be something like this, but obviously this does not work.

For those who, like me, didn't see the problem at first, the error is:

error: constant expression depends on a generic parameter
   |
   |                     let mut a = [0; T::my_byte_size];
   |                                     ^^^^^^^^^^^^^^^
   |
   = note: this may fail depending on what value the parameter takes