Best way to do callback with passed-through object

[John_Nagle]

I have a callback situation, and would like to know the best way to do it in Rust.

fn callback(decoder: &mut Msgdecode) -> Result<(), &'static str> { ... needs to access decoder....}

let decoder = Msgdecode{}; // caller's decoder object

let result = decode(&mut decoder, callback);

The plan is that the "decode" fn will be in a library crate, and used by various programs which have their own "decoder" structs of different types. Right now, the decoder struct has to be of type Msgdecode. I'd like that to be generic.

"decode", though, calls into 10,000 lines of machine-generated code that has to pass the Msgdecode struct down through layers of calls until it reaches the callback. So there's a huge generic instantiation involved if done that way. "decode" never actually does anything with the Msgdecode struct; it just passes it through to the callback function provided by the caller.

Why use a callback? Because the callback is called with a large borrowed stack-based data structure. Can't return a temporary, but you can let the callback borrow it and look at it.

Is there a better way to do this? Use a generic anyway? Traits? Rust has no equivalent of "pointer to function member" as far as I know.

[Michael-F-Bryan]

What about using dynamic dispatch instead of compile-time generics? So your function would accept a &dyn Fn(...) instead of being generic over F: Fn(...).

[John_Nagle]

I'm trying, but I can't figure out quite how to do that. The problem is that "self" inside functions of the trait can't be passed around as a MessageDecoder, or even a dyn MessageDecoder. Error message is

error[E0277]: the trait bound `&mut Self: traitpractice2::MessageDecoder` is not satisfied
  --> src/traitpractice2.rs:10:21
   |
10 |         moredecoder(&mut self,s);       // how to get "self" into impl Msgdecoder
   |                     -^^^^^^^^
   |                     |
   |                     the trait `traitpractice2::MessageDecoder` is not implemented for `&mut Self`
   |                     help: consider removing the leading `&`-reference
...
16 | fn moredecoder(dc: &mut impl MessageDecoder, s: &str)
   |                              -------------- required by this bound in `traitpractice2::moredecoder`

Code:

// Message Decoder is the common trait for all users of the decoder system.
trait MessageDecoder {

    fn callback(&self, s: &str);    // this is overrriden
    fn decode(&mut self, s: &str)   // this is implemented here
    {   println!("In MsgDecoder decode, msg {}",s);
        moredecoder(&mut self,s);       // how to get "self" into impl Msgdecoder? ***WILL NOT COMPILE***
    }
}


//  Function which is not in any struct but passes through a MessageDecoder
fn moredecoder(dc: &mut impl MessageDecoder, s: &str)
{
    println!("In moredecoder, msg {}",s);
    dc.callback(s);
}



// PacketLogDecoder is a user of the decoder system
struct PacketLogDecoder {
}

// Implement trait Message Decoder for Packet Log Decoder to allow callback.
impl MessageDecoder for PacketLogDecoder {
    fn callback(&self, s: &str) {                   // or is this supposed to override
        println!("In PacketLogDecoder callback, msg {}",s);
    }
}

// ViewerDecoder is another user of the decoder system
struct ViewerDecoder {
}

//  Implement trait Message Decoder for Packet Log Decoder to allow callback.
impl MessageDecoder for ViewerDecoder {
    fn callback(&self, s: &str) {                   // or is this supposed to override
        println!("In ViewerDecoder callback, msg {}",s);
    }
}


#[test]

fn testdecoder()
{
    let mut testobj1 = PacketLogDecoder{};
    testobj1.decode("Msg 1");
    
    let mut testobj2 = ViewerDecoder{};
    testobj2.decode("Msg 2");
}

So the idea is that trait "MessageDecoder" is implemented for PacketLogDecoder" and "ViewerDecoder". That works. "decode" can be called on structs of those impls. That works. I want "decode" to call "moredecoder" which represents a huge piece of code outside the trait def. It can't use the trait reference directly but should be able to pass it forward to the callback. "moredecoder" compiles fine.

The problem is that I can't get "self" in decode into an explicit MessageDecoder so I can call "moredecoder". "self" in decode is a trait reference which can be a PacketLogDecoder or a ViewerDecoder, so that ought to work. Somehow. Advice?

[Yandros]

In your last snippet, in decode, self is already an &mut reference (to something that impl MessageDecode and that may be unsized (such as dyn MessageDecode), i.e., something that is ?Sized.

If you then do &mut self, you are having a double &mut indirection, and that's what was causing the problem:

trait MessageDecoder {
    fn callback (self: &Self, s: &str);    // this is overrriden
    fn decode (self: &mut Self, s: &str)   // this is implemented here
    {
        println!("In MsgDecoder decode, msg {}",s);
        moredecoder(self, s);
    }
}

//  Function which is not in any struct but passes through a MessageDecoder
fn moredecoder (dc: &mut (impl ?Sized + MessageDecoder), s: &str)
{
    println!("In moredecoder, msg {}",s);
    dc.callback(s);
}

• or fn moredecoder (dc: &mut (dyn MessageDecoder), s: &str)

[John_Nagle]

Thanks. I knew it shouldn't be a ref, but then I got 'size' errors. Those come with a good error message, suggesting "(impl MessageDecoder + ?Sized)"

The documentation on this is rather vague. Is there better info available?

trait MessageDecoder {

    fn callback(&self, s: &str);    // this is overrriden
    fn decode(&mut self, s: &str)   // this is implemented here
    {   println!("In MsgDecoder decode, msg {}",s);
        moredecoder(self,s);       // how to get "self" into impl Msgdecoder
    }
}


//  Function which is not in any struct but passes through a MessageDecoder
fn moredecoder(dc: &mut (impl MessageDecoder + ?Sized), s: &str)
{
    println!("In moredecoder, msg {}",s);
    dc.callback(s);
}

[Yandros]

Maybe you find this post helpful:

What's the difference between T: Trait and dyn Trait?

<T : Trait> means that the function / trait / etc. can be "fed" / "instanced with", at compile time, any type T as long as T implements Trait. For each different T, a different version of the code will be generated / copy-pasted. dyn Trait, on the other hand, represents only one single concrete type, the one that unifies "all" the (other) T : Trait. Indeed, any (slim) pointer to a T when T : Trait can be coerced into a (fat) pointer to the type dyn Trait. And the type dyn Trait has an autom…

---

Basically, if you have some concrete non-slice type, you are dealing with a Sized type (T : Sized must be read as: all instances of type T have the same, statically-known / known-at-compile-time, size (this allows passing such instances by value, without needing pointer indirection)).

• Currently, the two exceptions to this Sized property are:

  • slices: [Item] (where Item : Sized, btw) and str. These types represent spans of memory whereby some instances of this type may be bigger or smaller than others.

  • type-erased "objects" = trait objects = dyn traits.

    This is when you have instances of different types but with same behavior (e.g. closures). Through pointer indirection, for any type T : Sized + Trait, a pointer to that T, such as a Box<T> can be coerced to a pointer to this type-erased type, dyn Trait, such as Box<dyn Trait>. The pointee of Box<dyn Trait>, dyn Trait, is now a single opaque type that represents all the possible types T : Trait. Because of that, it cannot be (statically) Sized, since these <T : Trait> types do not necessarily all have the same size.

In practice

• All generic parameters are implicitly Sized

  • including impl Trait syntax sugar.

• except for the Self type inside a trait definition.

This is why ?Sized "unbound" (remove the implicit Sized bound) must then be used on these generics, so that they are maximally generic, and do cover unsized objects such as slices or trait objects.

This is also why, some traits with default implementations need to add a : Sized bound on the trait itself, or a where Self : Sized clause to that method with a default implementation.

---

In your case:

 trait MessageDecoder {
    fn callback(self: &Self, s: &str)
    ;
    fn decode(self: &mut Self, s: &str)
    { // default implementation of a method inside a trait:
      // What we know / Rust knows here is that:
      //   - `Self : MessageDecoder`
      //   - `Self : ?Sized` (default for this case)
        println!("In MsgDecoder decode, msg {}", s);
        moredecoder(self, s);
    }
}

fn moredecoder(dc: &mut (impl MessageDecoder /* + Sized */), s: &str)
//                                          ^^^^^^^^^^^^^^
//                                         implicit default 
{
    println!("In moredecoder, msg {}",s);
    dc.callback(s);
}

Hence your original error.

So, with what we have seen, there are two possibilities to solve this issue:

• Add a Sized bound somewhere within the trait or the default method:

  • on the trait itself:

    - trait MessageDecoder {
    + trait MessageDecoder : Sized {
    

    This solves the issue, but forbids any usage whatsoever of dyn MessageDecoder (dyn Sized is absurd / an oxymoron).

  • on the default method:

      fn decode(self: &mut Self, s: &str)
    + where
    +     Self : Sized,
      {
          ...
      }
    

    This also solves the issue, but it makes it so dyn MessageDecoders cannot call .decode() anymore (they can call other non-where Self : Sized-annotated methods, though).

• Loosen the implicit Sized requirement on moredecode:

  - fn moredecoder(dc: &mut (impl MessageDecoder /* + Sized */), s: &str)
  + fn moredecoder(dc: &mut (impl MessageDecoder + ?Sized     ), s: &str)
  

[John_Nagle]

Ah. I didn't understand that "self" in a trait was automatically "dyn". That makes sense, though; that's the whole point of traits.

Anyway, here's a short version of what I ended up with, now including all the lifetime annotations. What's going on here is un-marshalling structs from a big slice of bytes, as a library. So different users need to call into this message decoder with a slice of bytes b, and get a decoded message back as a tree of temporary structs. Hence the callback. Some decoded structs have references to the slice of bytes b, because it contains blobs that need to be passed back to the callback and shouldn't be copied unnecessarily. The goal is to un-marshall without heap allocation.

Comments?

//
//  Trait practice
//
///  DecoderUser is the common trait for all users of the decoder system.
///  Callers must implement DecoderUser for their type so that they
///  will get a callback.
///
trait DecoderUser {
    //  Call this to decode.
    fn decode(&mut self, msgnum:  u32, b: &[u8], offset: &mut usize) -> Result<(), &'static str> {
        decodeswitch(self, msgnum, b, offset)    // fan out on msgnum
    }
    //  Be called back via this.
    fn decodedresult(&mut self, ms: &AllMessages) -> Result<(), &'static str>; // The callback
}

pub struct TestMessage{ test0: u32 }
pub struct PacketAck<'a>{ pub name: &'a [u8] }

pub enum AllMessages<'a> {
    MsgTestMessage(TestMessage),
    MsgPacketAck(PacketAck<'a>)
}
/// Switch which calls all decoders based on message number.
fn decodeswitch<'a,'b>(decoderuser: &mut (impl DecoderUser + ?Sized + 'b), msgnum:  u32, b: &'a[u8], offset: &mut usize) -> Result<(), &'static str>  {

    match msgnum {  // fan out on message number and call appropriate decoder.
        0xffff0001 => decode_test_message(decoderuser, b, offset),
        0xfffffffb => decode_packet_ack(decoderuser, b, offset),
        _ => Err("Unimplemented msg number")
    }
}

fn decode_test_message<'a>(decoderuser: &mut (impl DecoderUser + ?Sized), b: &'a [u8], offset: &mut usize) -> Result<(), &'static str> {
    let result = AllMessages::MsgTestMessage(TestMessage{test0: 1}  );
    *offset = *offset + 1;
    decoderuser.decodedresult(&result) // do callback, return its result
}

fn decode_packet_ack<'a>(decoderuser: &mut (impl DecoderUser + ?Sized), b: &'a [u8], offset: &mut usize) -> Result<(), &'static str> {
    let result = AllMessages::MsgPacketAck(PacketAck{name: &b});
    *offset = *offset + 1;
    decoderuser.decodedresult(&result) // do callback, return its result
}

struct PacketLogDecoder {}
// Implement trait Message Decoder for Packet Log Decoder to allow callback.
impl<'a> DecoderUser for PacketLogDecoder {
    fn decodedresult(&mut self, ms: &AllMessages) -> Result<(), &'static str> {
        println!("In DecoderUser for PacketLogDecoder");
        Ok(())
    }
}

#[test]

fn testdecoder()
{
    let mut testobj1 = PacketLogDecoder{};
    let testdata = [1u8, 2u8];  // junk test data
    let msgnum = 0xffff0001;
    let mut offset: usize = 0;
    let result = testobj1.decode(msgnum, &testdata, &mut offset);
}