How to store callbacks with `dyn Deserialize`-ish?

dakom · July 8, 2022, 11:25am

I would like to store a callback that takes anything serializable, and returns something deserializable from that

My attempt so far is this, but the compiler yells at me:

struct Callbacks {
    pub query: Box<dyn Fn(&dyn Serialize) -> Box<dyn Future<Output=Box<dyn Deserialize>>>>
}

In case it matters, this is stored on a struct which impls async_trait without Send, like this:

struct Holder {
    pub callbacks: Arc<Callbacks>
}
#[async_trait(?Send)]
impl Foo for Holder {
    async fn call(&self) {
        let value = &self.callbacks.query("foo").await;
    }
}

steffahn · July 8, 2022, 11:28am

You could look into https://crates.io/crates/erased-serde

Edit: they don't seem to providing an object safe version of Deserialize there, only Serialize as well as the *-er-traits (Deserializer, Serializer). I haven't yet thought through the question whether that's some principle limitation and/or how that might be or not he a problem for your use-case.

Edit2: Well, on second thought, of course there's some sense to this, since Deserialize::deserialize doesn't involve any self parameter. You might want to clarify more on what your use-case is.

VorfeedCanal · July 8, 2022, 11:46am

It's probably not an issue with topicstarter's case, but he would need to wrap Deserialize into something. Since usually deserialize returns Result<Self, …> which is not possible with dyn Trait.

Whether it would require the whole different Serde-like crate or not is open question.

P.S. The more I see questions like that the more I like how Swift solved that. How hard would it be to do something like that in Rust, I wonder?

dakom · July 8, 2022, 12:08pm

Basically, writing a generic layer in Rust that will abstract over some communication things in different environments (wasm, binary, etc.) - so it needs to be agnostic over the network layer itself, and let that be described as async callbacks.

This seems to compile... still have lots of other errors to tackle, so not sure if the whole thing really works yet, but it's a step in the right direction I think:

pub struct Holder {
    callbacks: Callbacks,
}

struct Callbacks {
    pub query: Box<dyn Fn(&str, Box<dyn erased_serde::Serialize>) -> Pin<Box<dyn Future<Output=Box<dyn erased_serde::Deserializer>>>>>,
}

impl Holder {
    pub async fn query<M: Serialize + 'static, R: DeserializeOwned + 'static>(&self, target: &str, msg: M) -> R {
        let mut value = (self.callbacks.query) (target, Box::new(msg)).await;

        erased_serde::deserialize(&mut value).unwrap_ext()
    }
}

#[async_trait(?Send)]
impl Foo for Holder {
    async fn foo(&self) -> MyDeserializableThing {
        self.query("bar", MySerializableThing).await
    }
}

steffahn · July 8, 2022, 12:21pm

Okay, I mostly wondered about qualifications what "something deserializable" means in

"something serializable" is clear, that's a value that can be serialized, but "deserializable" is usually a property of a type, not a value. If you already have the value, you don't need to deserialize it anymore. The dyn Deserializer you have now is rather "something you can (try to) deserialize into a value (of any type you like)", but perhaps that's what you wanted in the first place. And the code you posted looks reasonable. Edit: On second read of my own description here, "something deserializable" is l not actually a bad summary in there first place, since after all the Deserializer is the input to deserialization. Always takes a while to wrap one's head around all those serde traits again after a while; it's like a little Lego puzzle where you have to connect all the pieces, but once they fit and the compiler doesn't complain anymore, there's a good change one has built something sensible.

Alright, feel free to ask any further questions at a later point in case they come up.

H2CO3 · July 8, 2022, 1:18pm

I think it is, and it's more fundamental than "Self would need to be Sized". Even if by-value unsized returns were possible, it would still not make much sense to have a dyn Deserialize. After all, dyn means that "I already have an object of some type, but it might change dynamically, so I am hiding the concrete type". But deserialization contradicts both points:

there is no object, because we are trying to make the object; and
the type can't be ~~"dynamic"~~ existential, because the exact mechanism of deserializing into an object depends on the concrete type. So the only way to get back a deserializable existential would be to first deserialize into a concrete type (that has to be provided), and then convert it into a trait object, but that would be mostly pointless, because this implies that a concrete type had to be known beforehand anyway.

VorfeedCanal · July 8, 2022, 3:27pm

It's mostly pointless in today's Rust because there are no polymorphic implementation of traits and no support for the dynamic linking.

It's the other way around: Rust would need to support Swift-style polymorphic (not monomorphic like today) implementation of generics and the ability to put dynamically-sized types on stack.

In Rust (in contrast to C++) it's actually possible in theory (because it doesn't have move constructors, every type can be moved with memcpy and it doesn't support calling arbitrary functions and you can not call functions not provided by trait constraints, either), but would require many man-years of work (someone would need to cleanup and port Swift-related LLVM changes to master LLVM branch, e.g.).

But in that case, yes, something like dyn Deserialize would be useful: actual type would only be known to callback (presumably loaded from plugin) while the rest of the code could just use appropriate trait.

dakom · July 8, 2022, 3:39pm

ok, well, I painted myself into a corner somehow... what I have above is not compiling all the way

So, fine, dyn Deserializer is not what I wanted, rather I want dyn Deserialize, but I don't see how to store that yet

H2CO3 · July 8, 2022, 3:41pm

I'm not sure what you are referring to here; what I wrote is true as-is.

dakom · July 8, 2022, 3:49pm

This also partway compiles... guess I'll try it out (using Any as a placeholder for the Deserialized)!

EDIT: nope, it fails... added in a more complete example

137 | query: Box::new(actual_query)
| ^^^^^^^^^^^^ the trait for<'de> Deserialize<'de> is not implemented for (dyn std::any::Any + 'static)

Tried to put this on playground, but erased_serde isn't in the top 100 ...

Going to step away for a day or so. Maybe it'll all make sense when I get back

use std::future::Future;
use std::pin::Pin;
use std::any::Any;
use serde::{Serialize, Deserialize, de::DeserializeOwned};
use async_trait::async_trait;


fn main() {
    let holder = Holder {
        callbacks: MyCallbacks {
            query: Box::new(actual_query)
        }
    };
}

#[derive(Serialize, Deserialize)]
struct MySerializableThing {}

#[derive(Serialize, Deserialize)]
struct MyDeserializableThing {}

#[async_trait(?Send)]
trait Foo {
    async fn foo(&self) -> MyDeserializableThing;
}

pub struct Holder {
    callbacks: MyCallbacks,
}

struct MyCallbacks {
    pub query: Box<dyn Fn(Box<dyn erased_serde::Serialize>) -> Pin<Box<dyn Future<Output=Box<dyn Any>>>>>,
}

impl Holder {
    pub async fn query<M: Serialize + 'static, R: DeserializeOwned + 'static>(&self, msg: M) -> R {
        let value = (self.callbacks.query) (Box::new(msg)).await;

        *value.downcast().unwrap_ext()
    }
}

#[async_trait(?Send)]
impl Foo for Holder {
    async fn foo(&self) -> MyDeserializableThing {
        self.query(MySerializableThing{}).await
    }
}

fn actual_query<M: Serialize + 'static, R: DeserializeOwned + 'static>(msg: M) -> Pin<Box<dyn Future<Output=Box<R>>>> {
    Box::pin(async move {
        Box::new(serde_json::from_value(serde_json::json!({})).unwrap())
    })
}

dakom · July 8, 2022, 4:18pm

oh boy... looks like I'm running into similar issues as this: Some abstractions cannot be implemented - #10 by Yandros

dakom · July 9, 2022, 7:00pm

Ok... so I think I can reframe my problem to be more helpful. Tbh above I was kinda copy/pasting/editing from my code without giving much thought to what the actual problem is at a higher level.

The real issue is I have something like this being provided from downstream:

async fn foo<A: Serialize, B: DeserializeOwned>(args:A) -> B { ... }

and I want to store it in my library's struct, as a callback to be called later.

How do I do that?

(I am thinking maybe I actually just store it as a Boxed Fn with Any as both args and returns... but... curious to hear if there's a better way)

steffahn · July 9, 2022, 8:59pm

If you can generalize the signature to

async fn foo<A: Serialize, B, S: for<'a> DeserializeSeed<'a, Value = B>>(args: A, seed: S) -> B {
    todo!()
}

then it should become possible:

use std::{future::Future, pin::Pin};

use serde::{de::DeserializeSeed, Serialize};

use m::*;

#[allow(unused_variables)]
async fn foo<A: Serialize, B, S: for<'a> DeserializeSeed<'a, Value = B>>(args: A, seed: S) -> B {
    todo!()
}

struct Foo;

impl Callback for Foo {
    fn call<'lt, A: 'lt, B: 'lt, S: 'lt>(
        &'lt self,
        args: A,
        seed: S,
    ) -> Pin<Box<dyn Future<Output = B> + 'lt>>
    where
        A: Serialize,
        S: for<'a> DeserializeSeed<'a, Value = B>,
    {
        Box::pin(foo(args, seed))
    }
}

// demonstration:
async fn foo_indirectly<A: Serialize, B, S: for<'a> DeserializeSeed<'a, Value = B>>(args: A, seed: S) -> B {
    let erased: ErasedCallback<'static> = Box::new(Foo).into();
    erased.call(args, seed).await
}

async fn foo_indirectly_less_generic_usage<A: Serialize, B: DeserializeOwned>(args: A) -> B {
    foo_indirectly(args, PhantomData::<B>).await
}

// implementation:
mod m {
    use erased_serde as e;
    use serde as s;
    use std::{future::Future, pin::Pin};

    pub trait Callback {
        fn call<'lt, A: 'lt, B: 'lt, S: 'lt>(
            &'lt self,
            args: A,
            seed: S,
        ) -> Pin<Box<dyn Future<Output = B> + 'lt>>
        where
            A: s::Serialize,
            S: for<'a> s::de::DeserializeSeed<'a, Value = B>;
    }

    pub struct ErasedCallback<'lt>(Box<dyn DynCallback + 'lt>);

    trait DeserializeSeedInPlace<'a> {
        fn deserialize_in_place(&mut self, d: &mut dyn e::Deserializer<'a>)
            -> Result<(), e::Error>;
    }
    struct DeserializationWrapper<D, S>(Option<S>, Option<D>);
    impl<'a, D, S: s::de::DeserializeSeed<'a, Value = D>> DeserializeSeedInPlace<'a>
        for DeserializationWrapper<D, S>
    {
        fn deserialize_in_place(
            &mut self,
            d: &mut dyn e::Deserializer<'a>,
        ) -> Result<(), e::Error> {
            self.1 = Some(self.0.take().unwrap().deserialize(d)?);
            Ok(())
        }
    }
    impl<'de> s::de::DeserializeSeed<'de> for &mut dyn for<'a> DeserializeSeedInPlace<'a> {
        type Value = ();

        fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
        where
            D: s::Deserializer<'de>,
        {
            self.deserialize_in_place(&mut <dyn e::Deserializer>::erase(deserializer))
                .map_err(<D::Error as s::de::Error>::custom)
        }
    }

    trait DynCallback {
        fn dyn_call<'lt>(
            &'lt self,
            args: &'lt dyn e::Serialize,
            seed_and_result_slots: &'lt mut dyn for<'a> DeserializeSeedInPlace<'a>,
        ) -> Pin<Box<dyn Future<Output = ()> + 'lt>>;
    }

    impl<C: Callback> DynCallback for C {
        fn dyn_call<'lt>(
            &'lt self,
            args: &'lt dyn e::Serialize,
            seed_and_result_slots: &'lt mut dyn for<'a> DeserializeSeedInPlace<'a>,
        ) -> Pin<Box<dyn Future<Output = ()> + 'lt>> {
            Box::pin(self.call(args, seed_and_result_slots))
        }
    }

    impl Callback for ErasedCallback<'_> {
        fn call<'lt, A: 'lt, B: 'lt, S: 'lt>(
            &'lt self,
            args: A,
            seed: S,
        ) -> Pin<Box<dyn Future<Output = B> + 'lt>>
        where
            A: s::Serialize,
            S: for<'a> s::de::DeserializeSeed<'a, Value = B>,
        {
            Box::pin(async move {
                let mut slots = DeserializationWrapper(Some(seed), None);
                self.0.dyn_call(&args, &mut slots).await;
                slots.1.unwrap()
            })
        }
    }

    impl<'lt, C: Callback + 'lt> From<Box<C>> for ErasedCallback<'lt> {
        fn from(x: Box<C>) -> Self {
            ErasedCallback(x)
        }
    }
}

This compiles, I haven’t tested it yet though, because I didn’t have any test case implementation of foo.

dakom · July 10, 2022, 6:04am

Yikes... I mean, I really appreciate it - genuinely, thank you!

But I have to say this seems very absurd.

Storing a callback with strongly types parameters should not be this difficult!

H2CO3 · July 10, 2022, 6:13am

No, you don't – that's what we have been explaining all along.

The problem is, your callbacks aren't typed if you want to store them as "function from any serializable to any deserializable". This is effectively dynamic typing. And the problem is, again, that this loses the type information needed for deserializing. The trick is that the seed, albeit type-erased, is an object, so you can at least store some dynamic type information at the value level, which can initiate the (in-place) deserialization.

dakom · July 10, 2022, 6:38am

What I want is definitely dyn Deserialize. I reluctantly accept that I can't have that though (in Rust... which is a tough pill to swallow since this is super easy bread-and-butter stuff in Typescript, Go, and other languages - which suck in other ways).

steffahn · July 10, 2022, 7:52am

FYI, there's a fundamental problem with capturing the original signature

async fn foo<A: Serialize, B: DeserializeOwned>(args:A) -> B { ... }

in a dynamic value: in this function, the type B comes with a description of how its deserialized in the Form of the DeserializeOwned impl, but this description is compile-time information only, there's no way to provide any of the information at runtime! Note that, whatever we do, such a function can only be instantiated / monomorphized a finite number of times, from which we then want to recover its whole behavior essentially. This can of course be made work for a finite list of concrete types by using Any and a call to foo for each of finite many types for B, but that always only captures part of the behavior of foo.

The signature with DeserializeSeed however does contain a runtime-argument that can be used to describe the data format at runtime.

Another alternative would have been to call, foo with a general represention type of serde values, i. e. something like this type. This however only works if foo operates on "self-describing" data formats only (i. e. ones that support Deserializer::deserialize_any), whereas with the seed approach, we can keep supporting all data formats including compact binary ones.

Note that the way foo's behavior is captured in my code is also special w. r. t. the return value. It let's the function return via the seed parameter and makes B be (); this should not be a problem, because the only way a generic function like this can produce (and thus return) a B is by deserializing something on the given seed. Furthermore, this process consumes the seed, so it can only deserislize once and it can't do anything else to the return value but return it unmodified; at lease until we might eventually have something like specialization, in which case it would become truly impossible to capture the "real" behavior of a function foo like that down to the way it might be specializing on the choice of return type or argument type, in case foo makes some use of that. Well, actually, even right now if foo made shenanigans like different behavior depending on the size_of::<B>(), that would not be reproduced in foo_indirectly.

Perhaps this context gives you more of a feeling where the problems lie in "storing a generic function" like that.

Cerber-Ursi · July 10, 2022, 7:52am

Could you explain what do you mean here? When we have the already-working example, we'll probably understand better what exact semantics do you want - that's probably not the dyn Deserialize.

steffahn · July 10, 2022, 8:05am

To give some more context, the main problem you need to be working around is that Rust used monomorphization to implement generic functions. Many other strongly typed languages don't (including typescript, or for example Java or Haskell; you mention Go, but they ~~don't~~ didn't ^[1] even have actual generics in the first place), so a generic function is still just a single function at run-time, not a whole list of monomorphized instances that's infinite in principle and can only be made finite because your program using it is finite and your compiler prevents polymorphic recursion and doesn't let you store generic functions in a data structure.

E. g. Haskell, which uses “dictionary-passing” as an implementation technique instead of monomorphization, makes storing a generic function in a data structure trivial, i. e. it will just work out of the box, since everything is always working with dyn Trait-like values under the hood anyways.

they do now, right? And a quick web search suggests they do use monomorphization, so perhaps your perception that something like this can never be a problem in Go is outdated once their generics come into play!? ↩︎

H2CO3 · July 10, 2022, 8:29am

_{I think you meant DeserializeSeed here.}

Topic		Replies	Views
Async callback with Serde Deserialize (Zero-copy) help	3	655	June 11, 2023
Async callbacks in Rust	4	1355	May 2, 2022
Store async callback help	4	1071	March 14, 2022
What are reasonable ways to store a callback in a struct? help	7	10447	January 12, 2023
Generic parameter bound by `Deserialize<'de>`. Is this pattern possible in Rust? help	5	1054	February 26, 2023

How to store callbacks with `dyn Deserialize`-ish?

Related Topics