Convert between MANY library enum types

Hey,

I have an architectural question. I currently write a software which handles a lot of input streams (warp-API, kafka, aws kinesis, redis pub/sub, mqtt, ...), does some crazy calculations with database caching in mongo, redis, postgres and then returns to the same or another stream.

Now after implementing some inputs and databases, I see that many of the libs I use have their own typedefinition enums such as serde_json::Value, bson::Bson, evalexpr::Value and redis::Value. Some have automated conversion to the other (serde and bson), some don't. What would be the idiomatic way to handle that amount of different datatypes, because as far as I can tell from the compiler, I am not allowed to simply implement From for 2 library enums, right? I implemented a lot of utilities functions like from_evalexpr_val_to_serde_json_val(val: evalexpr::Value) -> serde_json::Value. But this I really dislike like because I'm not a huge fan of utility modules in ANY language. This was more due to the fact that there was a deadline coming and I didn't have time to think about this problem by then.

  1. My current plan would be, to implement my own internal enum I use for representing the way I want to handle MY data, and then write From implementations for each library I use when I add another input stream or database.

  2. Another thing I'm thinking about could be possible but I don't really understand is is writing serde Serializers and Deserializers for all crates which do not support it, and then use serde to convert between those. As mentioned, for me it's more like a gut feeling that this could be possible, but I don't really understand how. If, maybe someone can give me a short pointer on how I would implement this.

  3. ??? Maybe there's something I don't see, but I want to write the code as idiomatic as possible.

best, Tobi

Both 1 and 2 are reasonable. My preference would be to normalize around one particular structure, ideally something that has maximum portability. The serde crate is quite popular, so it's not a bad place to start. Ideally you want to minimize the number of type conversions you have to perform.

Another thing to note: you can make your life easier by writing your interfaces to accept parameters in the form of var: impl Into<T>. For example, if you have a function that you want to accept a string for, you can do something like this:

fn my_func(string_param: impl Into<String>) {
  // ...
}

In other words, you can specify the parameter by merely saying "this function accepts anything that can be Into'd as a String. Then all you need to do is provide the From implementation for types that don't provide it.

It's preferable to coalesce around a specific concrete type. The downside to such type conversions is that you may need to copy a lot of bytes around. And while the JSON-like enums (such as serde_json::Value), they do create a little more work because you have to pattern match on them every time you need to extract data, whereas with a plain old struct you can skip this.

2 Likes

Thanks for the detailed reply! In general I want to say that I really appreciate how the rust community handles questions in this forum or discord!

Even in the short time I'm currently working on rewriting one of our core services in Rust I was able to learn a lot about how we could have implemented things easier in other languages we used before, especially C++.

Regarding the var: impl Into<T> pattern, I discovered that when reviewing code of other projects since I started, but there are definitely a hands full of functions which could make good use of that but I have implemented prior to finding out about this. So thanks again for reminding me about that =)!

1 Like

There's nothing terribly wrong with this, but you could organise all those into a trait and trait implementations.

Essentially just work around the orphan rule not by making your own type, but by making your own trait:

pub trait MyFrom<T> {
    fn from(other: T) -> Self;
}

impl MyFrom<evalexpr::Value> for serde_json::Value {
    fn from(val: evalexpr::Value) -> serde_json::Value {
          from_evalexpr_val_to_serde_json_val(val)
    }
}

// You can add MyInto and a blanket impl too if you like!

Disclaimer: It's decidedly not idiomatic. The idiomatic way to work around the orphan rule is to make newtype wrappers, not duplicate traits.

I wonder if the idiomatic approach would be less cumbersome (so many newtypes! so many variants!) by having a single generic struct:

MyValue<T> {
    value: T
}

// Or even just
MyValue(T);

And then implementing From<serde_json::Value> and the like for MyValue<serde_json::Value>.

Then you're free to implement From<MyValue<...>> for ... as you see fit.

Ah and if you impl Into<serde_json::Value> and the like for MyValue<serde_json::Value>, you could probably do a blanket impl of From MyValue X to MyValue Y.

1 Like

I tried to demo the blanket impl part it in the playground (I rushed to the end of my train of thought!) but it failed.

the blanket impl fails on recursive requirements (kinda makes sense).
Even a humble convert function seems to wrap itself again - I'm not sure why.

Annoyingly this makes a "generic from converter newtype" pretty unwieldly :confused:

1 Like

Hey @drmason13

Thanks for the detailed reply, I really appreciate that! We already went pretty far with the own type and also found other reasons why this makes sense for use, especially because we have reoccuring checks on different datatypes that we have. One example is that we need to accept timestamps as int and strings of different format, all in json and then using different calculations on it which are easier if you use a DateTime object. In our case for example we need to calculate day differences (24h) as well as calendaric day differences which are much handier, if you use DateTime anyway.

Anyways, I really appreciate you taking the time to write this down because I already have an Idea where I can use the "newtype" pattern together with generic functions. I currently have to learn rust "on the fly", that's why I usually see lots of refactoring potential on code I wrote 2 weeks ago based on less knowledge about the language. Thanks for giving me even more inspiration! :clap:

1 Like