Advice for API for file parsing library

code review
1

Hi all, I have a fairly broadly scoped question but I'm in a position where I know roughly what I want to do but no idea if it's possible in Rust or at all (it may be non-sensical). So, basically I have a file type which can contain many different types of data (f32, i8, u16 etc), as you read the file, meta-data tells you what data type the raw data is. I'm writing a library to read data, and I would like the user of the library to be able to just say "read channel a" and get back either a vector of data which they can use naively (I think this is impossible maybe?) or to get back something that can be used like a vector but hides all the "what type of data is this?" machinery, I ended up with an enum as follows:

pub enum DataTypeVec {
    Void(Vec<()>),      // Should nuke this somehow
    Boolean(Vec<bool>), // nptdms uses 1 byte, I'm not sure this is correct as LV internal representation is 32 bits for a bool
    I8(Vec<i8>),
    I16(Vec<i16>),
    I32(Vec<i32>),
    I64(Vec<i64>),
    U8(Vec<u8>),
    U16(Vec<u16>),
    U32(Vec<u32>),
    U64(Vec<u64>),
    Float(Vec<f32>),
    Double(Vec<f64>),
    //Extended(Vec<f128>), Can't represent this currently
    FloatUnit(Vec<FloatWithUnit<f32>>),
    DoubleUnit(Vec<FloatWithUnit<f64>>),
    //ExtendedUnit(Vec<FloatWithUnit<f128>>), Can't represent this
    TdmsString(Vec<String>),
    // DaqMx(Vec<??>)
    // ComplexSingle(Vec<??>)
    // CompledDouble(Vec<??>)
    TimeStamp(Vec<TdmsTimeStamp>),
}

But I haven't been able to figure out how to provide the machinery to let the library users interact with this enum as if it were just a vector. Things I can imagine the user wanting to do:

  • Take the data and transform it into something they can plot (From implementations for each enum variant?)
  • Take it and convert it into their data format easily
  • Iterate over it and print

My ultimate goal is for the user to be able to read the data and do anything they might typically do on Vec where T is numeric but no idea how to express that in Rust. The files also log strings and you can have channels of strings but I'm ok with the library just sending back an error if the user tries to do numeric things with string data.

I have an implementation here: https://github.com/AJAnderson/rstdms

For the curious, a friend of mine has also been working on an implementation of the same thing (independently) and has come up with the following: https://github.com/adamreeve/rstdms.

He has a really nice mechanism for handling reading in a generic way, but ultimately he pushes the responsibility of providing an appropriately sized and typed buffer to handle the data onto the user. Ideally I'd like the user to be a bit more naive about the exact type of the channel they're asking for.

2

Not sure if this is considered bad etiquette but ping

3

You can implement your own iterator for the enum (and also IntoIterator so the enum can be used in for loops). The iterator item type would be:

enum DataType {
    I8(i8),
    I16(i16),
    // etc.
}

Note that in your next() method implementation you would probably have a bunch of repeated code for each enum variant, so you could generate the implementation with macro_rules!.

If you want the items to be printable you can add #[derive(Debug)] for printing with {:?} or implement Display for DataType for printing with {}.

To make DataType workable as numeric data you can probably impl TryFrom<DataType> for f64 or something like that.

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.