Serialisation of vector of struct to byte array


#1

Hi I just started out with Rust which I am really enjoying.
However, I am not sure how best to accomplish a serialisation task.

Basically I have a structure like

#[derive(Debug, Clone, Copy)]
struct SensorValueArray {
    id: i32,
    len: i32,
    values: Vec<SensorValue>
}

Where SensorValue is

#[derive(Debug)]
struct SensorValue {
    dt: NaiveDateTime,   // byte format i64 + u32 = 8 + 4 = 12 bytes
    value: i64        //  8 bytes 
}

NaiveDateTime is part of the chronos crate. I only need millisecond accuracy so could store the NaiveDateTime as a 13 digit integer.

I would like to convert SensorValueArray to and from a byte array for sending over a tcp socket.

So I guess the byte array format should be something like

<sensor_id_bytes><length of entries_in_bytes><val1_datetime_bytes><val1_value_bytes>val2_datetime_bytes><val2_value_bytes> ...

So for reading I have something like below but not sure how to efficiently read the bytes into a SensorValue vector.

impl SensorValueByteReader {
    pub fn from_byte_array(raw: &[u8]) -> Option<BmosSensorValueArray> {
        if raw.len() < 28 {
            None
        } 
        self.id = read_u32(raw);
        self.len = read_u32(raw[4..]);
        self.values = Vec<BmosSensorValue>::with_capacity(self.len);
        
        let mut rdr = Cursor::new(raw[8..]);
       // Not sure ho to read 
    }
}
fn read_u32(data: &[u8]) -> u32 {
    (data[0] as u32 << 24) +
    (data[1] as u32 << 16) +
    (data[2] as u32 <<  8) + 
    (data[3] as u32 <<  0)
}

Thanks for any advice on completing the serialisation to an from a byte array.


#2

Use Serde! Here is the complete code:

[dependencies]
bincode = "0.6"
chrono = "0.2"
serde = "0.8"
serde_derive = "0.8"
extern crate bincode;
extern crate chrono;
extern crate serde;
#[macro_use] extern crate serde_derive;

use chrono::NaiveDateTime;
use serde::{Serialize, Serializer, Deserialize, Deserializer};

#[derive(Debug, Serialize, Deserialize)]
struct SensorValueArray {
    id: i32,
    values: Vec<SensorValue>
}

#[derive(Debug, Serialize, Deserialize)]
struct SensorValue {
    #[serde(serialize_with = "dt_as_tuple", deserialize_with = "dt_from_tuple")]
    dt: NaiveDateTime,
    value: i64
}

// NaiveDateTime serializes in string representation by default; serialize it as
// (seconds, nanos) tuple instead.
fn dt_as_tuple<S>(ndt: &NaiveDateTime, serializer: &mut S) -> Result<(), S::Error>
    where S: Serializer
{
    (ndt.timestamp(), ndt.timestamp_subsec_nanos()).serialize(serializer)
}

// NaiveDateTime deserializes from string representation by default; deserialize
// it from (seconds, nano) tuple instead.
fn dt_from_tuple<D>(deserializer: &mut D) -> Result<NaiveDateTime, D::Error>
    where D: Deserializer
{
    let (secs, nanos) = Deserialize::deserialize(deserializer)?;
    Ok(NaiveDateTime::from_timestamp(secs, nanos))
}

fn main() {
    let array = SensorValueArray {
        id: 0x01010101,
        values: vec![
            SensorValue {
                dt: NaiveDateTime::from_timestamp(0x0000030303030303, 0x04040404),
                value: 0x0505050505050505,
            },
            SensorValue {
                dt: NaiveDateTime::from_timestamp(0x0000060606060606, 0x07070707),
                value: 0x0808080808080808,
            },
        ],
    };

    let bytes = bincode::serde::serialize(&array, bincode::SizeLimit::Infinite).unwrap();
    println!("{:?}", bytes);

    let array: SensorValueArray = bincode::serde::deserialize(&bytes).unwrap();
    println!("{:#?}", array);
}

Here are the bytes you get as output:

  // array id
  1, 1, 1, 1,
  // array len
  0, 0, 0, 0, 0, 0, 0, 2,
  // first dt seconds
  0, 0, 3, 3, 3, 3, 3, 3,
  // first dt nanos
  4, 4, 4, 4,
  // first value
  5, 5, 5, 5, 5, 5, 5, 5,
  // second dt seconds
  0, 0, 6, 6, 6, 6, 6, 6,
  // second dt nanos
  7, 7, 7, 7,
  // second value
  8, 8, 8, 8, 8, 8, 8, 8
]

#3

Thanks so much. Serde looks great.

Although serde_derive is giving the error

error[E0463]: can’t find crate for proc_macro
–> /home/glenn/.cargo/registry/src/github.com-88ac128001ac3a9a/serde_derive-0.9.2/src/lib.rs:15:1
|
15 | extern crate proc_macro;
| ^^^^^^^^^^^^^^^^^^^^^^^^ can’t find crate

error: aborting due to previous error

Could not compile serde_derive.

I think I need to maybe update rust. I am running rustc 1.14.0 in Fedora 25


#4

It requires rustc 1.15.0 or higher which is currently the beta channel but will be released in 5 days (on Thursday). Until then, you can use rustup to install the beta version of 1.15.0, then upgrade to the real 1.15.0 when the release comes out.

$ rustup install beta
$ cd your-sensor-project && cargo +beta build

On Thursday when the release is out:

$ rustup update stable
$ cd your-sensor-project && cargo build