Serialize/Deserialize Array of Structs

help
1

I feel like this has probably been answered before, but I cannot find how to make this work.

Say I have a struct such as struct A { a: usize, b: usize } and I want to read from disk an array of exactly 128 of these structs. I'm using the bincode package and attempting as such:

        let entries :[A; 128] = bincode::deserialize(my_file).expect("Error");

I get the following error:

the trait `serde::Deserialize<'_>` is not implemented for `[bat::A; 128]`
    = help: the following implementations were found:
              <&'a [u8] as serde::Deserialize<'de>>
              <[T; 0] as serde::Deserialize<'de>>
              <[T; 10] as serde::Deserialize<'de>>
              <[T; 11] as serde::Deserialize<'de>>
            and 30 others

Looking into the serde code, I found the following:

array_impls! {
    1 => (0 a)
    2 => (0 a 1 b)
    3 => (0 a 1 b 2 c)
... snip ...
    30 => (0 a 1 b 2 c 3 d 4 e 5 f 6 g 7 h 8 i 9 j 10 k 11 l 12 m 13 n 14 o 15 p 16 q 17 r 18 s 19 t 20 u 21 v 22 w 23 x 24 y 25 z 26 aa 27 ab 28 ac 29 ad)
    31 => (0 a 1 b 2 c 3 d 4 e 5 f 6 g 7 h 8 i 9 j 10 k 11 l 12 m 13 n 14 o 15 p 16 q 17 r 18 s 19 t 20 u 21 v 22 w 23 x 24 y 25 z 26 aa 27 ab 28 ac 29 ad 30 ae)
    32 => (0 a 1 b 2 c 3 d 4 e 5 f 6 g 7 h 8 i 9 j 10 k 11 l 12 m 13 n 14 o 15 p 16 q 17 r 18 s 19 t 20 u 21 v 22 w 23 x 24 y 25 z 26 aa 27 ab 28 ac 29 ad 30 ae 31 af)
}

That seems to indicate to me that they only implemented arrays up to 32 items... so what is the best way to get 128 items? Do I need a custom Deserializer here? This seems like overkill and that maybe a simple mem::transmute might be the easiest/fastest way to go here.

Thoughts? Thanks!

2

I would simply deserialize into a Vec<A> instead.

If you really really must avoid allocating on the heap, you can manually implement Deserialize on a wrapper, and take out of the wrapper when you have deserialized it.

2 Likes
3

Until the const generics lands on stable, using arrays longer than 32 is pure pain in Rust. It's best to avoid them, don't expect them to work smoothly like "normal" types.

1 Like
4

@alice how would I go about using a Vec<A>? Somewhere I have to indicate that I only want to read 32 of the structs... where would I do that?

5

You can use Vec<A> like this:

let entries: Vec<A> = bincode::deserialize(my_file).expect("Error");

This will work for any length.

6

@alice it doesn't. The file is initialized with all zeros, and so it doesn't read any of them. Again, I need to tell the deserializer to stop after reading 32 of the structs, and I'm not sure how to do that. Thanks!

7

It sounds like the data was not serialized using bincode. Consider using the byteorder crate instead which provides a bunch of nice methods in the ReadBytesExt trait, which you can call on anything that implements Read which includes File (and the buffered version BufReader<File>).

8

Yea, it wasn't :expressionless: I thought it would be able to handle this fairly easily, but using a simple mem::transmute seems to be the easiest way to go here. In case someone else comes looking:

        let mut buff = [0u8; BLOCK_SIZE];

        // read in one block
        self.device.read_exact(&mut buff)?;

        // transmute the buffer to an array
        let entries :[A; ENTRIES_PER_BLOCK as usize] = unsafe { std::mem::transmute(buff) };
9

Note that the mem::transmute is undefined behavior because your A type is not #[repr(C)] and even if you add it, it now depends on the endianess of the system.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.