How to store callbacks with `dyn Deserialize`-ish?

Sure... here's an illustrative example of what I'm after in Typescript (this compiles and runs fine)

Again, I want to be very clear that I'm not really complaining! I am trying to understand. But this is pretty frustrating. Kinda up there with coming to Rust after working with strings anywhere else. I get it, Rust makes different tradeoffs and has very good reasons.

I further get that @steffahn has graciously provided examples of how to achieve pretty much what I'm after... it's not that this is impossible in Rust or even that it requires casting to raw pointers and back or something...

But the verbosity and complexity is still hella annoying by comparison!


// this interface doesn't care about the specific Serialize/Deserialize impls
interface AbstractCallbacks {
    abstract_query: (query:Serialize) => Promise<Deserialize>
}

// furthermore, this class doesn't care about it either
class Service implements AbstractCallbacks {
    constructor(public abstract_query: (query:Serialize) => Promise<Deserialize>) {
    }

    async login(req:AuthRequest):Promise<AuthResponse> {
        return await this.abstract_query(req);
    }
}

// this is the point - it's trivial to create these different service processors
// this is where they care about the specific implementation
const web_service = new Service(web_query);
const desktop_service = new Service(desktop_query);

async function web_query(query:Serialize):Promise<Deserialize> {
    //placeholder... real-world this might serialize and deserialize as JSON
    //and go through JS FFI stuff
    return query + ", response from web" as Deserialize
}

async function desktop_query(query:Serialize):Promise<Deserialize> {
    //placeholder.. real-world this might serialize and deserialize as Protobuf
    //and go through native TCP/UDP stuff, I dunno
    return query + ", response from desktop" as Deserialize
}

// to represent the stuff we want to guarantee matches up at compile-time
// of course real-world these would have various methods etc.
interface Serialize {
}

interface Deserialize {
}

interface AuthRequest extends Serialize {
}

interface AuthResponse extends Deserialize {
}

// proof that it works
web_service.login("request from web").then(console.log)
desktop_service.login("request from desktop").then(console.log)

Yeah... I'll probably end up doing something closer to this for starters, and then revisit your post here if/when I need to make it more flexible. thanks again!

heh, now that I started actually implementing things while accepting getting rid of the flexibility, it turned out to be much simpler... don't need boxing or trait objects... if I actually do need more flexibility I'll probably have an enum of CallbackKind

It's a bit much to post since it's a bit more involved (JsValues and extern things, not exactly serde's Value) - but really, thank y'all for the help!

I'd really appreciate any resources that help build a better intuition for how monomorphization, trait objects, and comparison to other languages plays out. I'll step through the Seed example more carefully later, hopefully... but this is not the first time storing callbacks w/ generics or trait objects has tripped me up. There's something that just hasn't quite clicked for me yet..

But it does care about the fact that Deserialize is, in fact, AuthResponse - it will be broken if the query passed to the constructor returns something else, won't it? That's the thing Rust forces you to think of at compile-time.

upd: In fact, once we make AuthRequest and AuthResponse non-trivial (that is, they are different types from Serialize and Deserialize respectively), the code in your post stops to compile, with a totally expected error - Promise<Deserialize> can't be assigned to Promise<AuthResponse>.

4 Likes

I'm not sure you would find many resources, because, ultimately, it's very-very simple and can be summarized as a very simple single sentence: generics don't exist in Rust-compiled binary.

That's it. Yes, generics do exist in the Rust source code, but before actual binary is emitted they are converted into sets of functions with different input and output types.

That's where that whole discussion went astray: you were looking on Rust source and tried to ask how can you store callbacks with generics… but of course you can not do that! Generics don't exist in binaries, means callabacks with generics don't exist either means you phrase about callbacks with generics have no meaning (when you look on it from existing Rust compiler).

And people were just trying to find out how exactly you plan to liquidate generics on the road to binary (I was talking about how Rust compiler can be modified to make generics actually exist at runtime instead — after such modification your question would have a meaning).

It's super-easy in Typescript, Go and other languages where generics do exist in runtime. Where generics are not just a fancy name for “a bunch of similar functions copy-pasted from a single template by compiler” but where they actually exist. When one, single (compiled-into-binary-single single) implementation works with diffrent types. And Rust, actually, can be extended to permit such an implementation (blog post of Niko Matsakis outlined how Swift have done that… Rust compiler can do the same).

But as long as generics have to disappear on the road to binary rules of Rust would be different! You can not store callbacks with generics anywhere in the compiled binary because such thing doesn't exist in said binary!

P.S. And yes, there is one exception from that rule in the existing Rust language: dyn Trait. But that one is closer to virtual functions in C++/Java than to generics: you can only use it to, basically, call bunch of simple, non-genericalized functions which may be implemented differently. Enough for OOP, not enough for tricks which you try to do.

No, query merely needs to return any kind of Deserialize, and it's completely agnostic to the specific type. In the example as-is, the type signature of the constructor and query implementations know nothing at all about "Auth".

Where it would break would be if the return from login returned some non-Auth type, and it would break at runtime. This is exactly equivilent to unwrapping a serde parser. The closest Typescript equivilent would be something like GitHub - gcanti/io-ts: Runtime type system for IO decoding/encoding but that's outside the scope here.

Bottom line: the query methods here do not return AuthResponse. It merely returns any kind of Deserialize (which is the whole point).

That's not making it "non-trivial", that's breaking the contract completely and turning the example into something else entirely.

What I think you're missing in your assessment is that Typescript is structurally typed, so it's straightforward for non-trivial types to also satisfy Serialize/Deserialize, even if we don't explicitly name it.

For example, if we take off the extends Serialize/Deserialize from the Auth* stuff in the example above, it still compiles fine.

Yeah... it's dyn Trait which really trips me up... or maybe the specific dyn Trait as it applies to serde stuff. I'll update the title

And that's exactly the reason you can't just use them inside login - there's no guarantee that it will work.

So, you want to first deserialize into something (not into the AuthResponse), then try to convert the result into AuthResponse and return an error if this conversion fails? In this case, you might want to use serde_json::Value as an intermediate representation, and when the type is known, you can use its Deserializer implementation to deserialize it into the type you need.

2 Likes

I mean, I did exactly that and it compiles fine. But I think we're going off topic, just because something works in Typescript doesn't mean it's simple in (safe) Rust, in cases like this it's really apples and oranges!

Yeah - but I didn't want Holder to have to know about that at all (similar to how Service in the typescript example didn't need to know about it).

I wanted only the downstream query implementations to choose their intermediate value (compare to the typescript example where it could be JSON or raw TCP/UDP bytes, whatever is tucked in web_query/desktop_query), and Holder only needs to know that query returns "something Deserializable".

It's okay though. By refactoring to a different approach altogether I was able to work around my problem in a different way :slight_smile:

In this case, "something Deserializeable" is probably "some specific implementation of Deserializer, which is enough to hold everything you worry about". It might be not serde_json::Value exactly, that's just the first thing that pops to mind, but it will be some specific static type which can be pulled from whatever source you have and then later, when the exact expected type is known, converted to that type.

2 Likes

Hmm I think this is the crux of what I'm missing... I don't quite 100% see it, but I think I am starting to get a bit of a hazy picture.

In the monomorphized version of DeserializeOwned, it somehow provides some essential information serde needs.

In a trait object, that information would be lost, but it can alternatively be provided via the associated type Value in DeserializeSeed

I'm still missing some steps - but is this roughly in the ballpark? If so, then seeing how the monomophization works to provide that "essential information" would probably fill in the gaps for me.

Yup, that's basically how I'm refactoring it! (Callbacks will hold the specific type, and I'll start with the "Web" version, if/when I need to add "Desktop" it means turning Callbacks into an enum with each variant having its specific type).

It's a bit of an inversion of what I wanted, it means the upstream library crate has to be a little bit aware of the intermediate type (or place other restrictions like DeserializeSeed)...

But honestly, this is for such a minimal thing anyway... I've spent like 500% more time thinking and talking about it than actually writing or using it :joy: , and now I have working a battle plan - so off to implement!

Much appreciate the conversation, I definitely learned something here - thanks for the nudge!

1 Like

Actually, I was referring to the seed. See, in this signature

async fn foo<A: Serialize, B, S: for<'a> DeserializeSeed<'a, Value = B>>(args: A, seed: S) -> B {
    todo!()
}

there is a runtime argument seed: S which will (have to) be passed to a call to DeserializeSeed::deserialize in order to produce a value of type B; but the deserialize function can thus use that seed value to influence deserialization.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.