What is the best practice to handle dynamic records?

Consider the following problem:

Our program need to handle dynamic records, the schema (types of the fields) of which is given at runtime.

The program will first get a vector of strings which specifies types of dynamic records as input, e.g.:

["i32", "String", "f64", "bool"]

then the program will receive dynamic records (as bytes stream), e.g.:

[1, "hello", 2.78, true],
[100, "world", 3.14, false],
[1000, "!!!", 42.0, false],

What's the best way to store the dynamic record? From my experience, the best approach requires dependent types which don't exist in Rust, and what's the alternative way in Rust?

Maybe you should just handle the value as byte streams and treat them according to the "type".

1 Like

"Dynamic" actually just means "one type".

So you just store it as a Value in serde_json - Rust and don't get help doing it correctly.

This approach seems to sacrifice the memory-efficiency :thinking:

I don't think even dependent types can solve this. If you want arbitrary types to be handled at runtime, then 1. it can't be statically typed, and 2. you would need to generate memory layouts and de/serialization implementations dynamically.

It seems unlikely to me that this is, in its full generality, should be a goal or requirement. Why do you think you need this, what are you trying to achieve?

Not necessarily. As you can see, any primitive value (number, boolean, string) is stored inline (well, string characters themselves are on the heap, but they would be there anyway), and extra indirection is necessary only for complex objects and arrays, which, again, are unlikely to be handled better manually.

Hello, I just need to handle limited types, i.e. i32, i64, f32, f64, bool, char, String and don't have to handle arbitrary types...

If that's the case, just define an enum with the variants you want.

enum Value {
  I32(i32),
  I64(i64),
  F32(f32),
  F64(F64),
  Bool(bool),
  Char(char),
  String(String),
}

In terms of memory usage, the memory usage will be the size of the largest variant plus the size of a tag and any padding needed to maintain alignment. There are also optimisations that take advantage of things like how a String contains a pointer that will never be null so that could be used to indicate something else, and so on.

This is essentially what serde_json::Value does, except it includes lists and hashmaps so you can get arbitrarily nested values.

4 Likes