Cyclic structs with Traits

Hello,

I have been working 2 weeks on cyclic references for a side project I am working on. So I have managed to have cyclic references and avoiding Mutex (I am on tokio) using Arc's nightly new_cyclic. Here is an mvp example:

#![feature(arc_new_cyclic)]
use std::sync::{Arc, Weak};

struct Builder {
    manager: Arc<Manager>
}

struct Manager {
    core: Arc<Core>,
}

struct Core {
    manager: Weak<Manager>
}

impl Core {
    pub fn say_hi(&self) {
        println!("hi!");
    }
}


fn main() {

    let manager: Arc<Manager> = Builder {
        manager: Arc::new_cyclic(|me| Manager {
            core: Arc::new(Core {
                manager: me.clone()
            })
        })
    }.manager;
    
    manager.core.say_hi();
}

rust playground link: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=22ba08bd3a6116b32e25696a086314c7

However, I want the "child" of the cyclic structure to adhere on a trait, so I moved over to that using dyn Trait:

#![feature(arc_new_cyclic)]
use std::sync::{Arc, Weak};

struct Builder {
    manager: Arc<Manager>
}

struct Manager {
    core: Arc<dyn CoreTrait>,
}

trait CoreTrait {
    fn say_hi(&self);
}

struct Core {
    manager: Weak<Manager>
}

impl Core {
    fn say_hello(&self) {
        println!("Say hello");
    }
}

impl CoreTrait for Core {
    fn say_hi(&self) {
        println!("say hi!");
    }
}


fn main() {
    let manager: Arc<Manager> = Builder {
        manager: Arc::new_cyclic(|me| Manager {
            core: Arc::new(Core {
                manager: me.clone()
            })
        })
    }.manager;
    
    manager.core.say_hi();
    manager.core.say_hello();
}

rust playground link: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=1618ebd8ef9eb09fb2b251c8e5143778

The problem now is that I completely lose the type information as you can see it doesn't compile. For me it's important to still keep the type and be able to access other parts of the type that is not defined in the trait.

I am trying to implement the same thing using generic types, but it's seems quite challenging. For starters, I have created an even more simple example here:

#![feature(arc_new_cyclic)]
use std::sync::{Arc, Weak};

struct Builder<T: CoreTrait> {
    manager: Arc<Manager<T>>
}

struct Manager<T: CoreTrait> {
    core: Arc<T>,
}

trait CoreTrait {
    fn say_hi(&self);
}

struct Core<T: CoreTrait> {
    manager: Weak<Manager<T>>
}

impl<T: CoreTrait> Core<T> where Core<T>: CoreTrait {
    fn say_hello(&self) {
        println!("Say hello");
    }
}

impl<T: CoreTrait> CoreTrait for Core<T> where Core<T>: CoreTrait {
    fn say_hi(&self) {
        println!("say hi!");
    }
}

fn main() {
    let core: Core<dyn CoreTrait> = Core {
        manager: Weak::new()
    };
}

rust playground link: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=ac404cfb0f81b92e415882da8118423a

While the types compile, main fails with an error saying the size for values of type dyn CoreTrait cannot be known at compilation time and some other things which don't help me to figure out what I need to change. Boxing things didn't make any difference.

I am a bit desperate, calling for help now, if anyone has any hint I would be more than grateful.

The first thing I did was remove the trait bounds in the structs. Generally it's better to only have them on the impl block.

struct Builder<T> {
    manager: Arc<Manager<T>>
}

struct Manager<T> {
    core: Arc<T>,
}

struct Core<T> {
    manager: Weak<Manager<T>>
}

Now you can just follow the compiler suggestions.

   = help: the trait `Sized` is not implemented for `dyn CoreTrait`
help: consider relaxing the implicit `Sized` restriction
   |
16 | struct Core<T: ?Sized> {
   |              ^^^^^^^^

giving

struct Manager<T: ?Sized> {
    core: Arc<T>,
}

struct Core<T: ?Sized> {
    manager: Weak<Manager<T>>,
}

The reason that you need this ?Sized bound is that by default all generic parameters include the bound T: Sized. To turn this implicit where bound off, you can do T: ?Sized.

The reason that this happens is because you almost always want the Sized bound. Without that bound, values of that type cannot exist without being behind some kind of pointer (in this case Arc).

Hi,

First of all, thank you for your reply! Indeed now it compiles. Regarding the ?Sized, why is needed, since Arc should be sized anyway. Is it due to the cyclic nature of the structs?

In case you are interested or it helps you, I can give you some context, so that the things I am trying to achive don't feel out of the blue.

I am trying to implement RFC3261 (SIP) in Rust. I have a repo here: https://github.com/vasilakisfil/webphone
RFC3261, defines 3 layers:

  • Transport layer, which is responsible receiving/sending messages from the socket
  • Transaction layer, which is responsible for filtering messages, taking care of acks or unecessary incoming provisional messages
  • Core layer, which is responsible for the basic logic

You can think each to be its own tokio task.
In theory when a message comes to the socket, Transport forwards it to Transaction, Transaction to Core and Core is responsible to do what it has to do.
However, in many cases each layer need to communicate more than one other layer for the same message.

For instance for the Transaction layer, when it receives a message, it could happen that it needs to forward it to both Transport and Core (with some small modifications).

The way I have implemented the basic architecture, is to store all 3 layers in an Arced SipManager like that:

pub struct SipManager {
    pub core: Arc<dyn CoreLayer>,
    pub transaction: Arc<dyn TransactionLayer>,
    pub transport: Arc<dyn TransportLayer>,
}

Of course in order to get the Arc<SipManager>, I am using a struct on top of it, named Builder:

pub struct SipBuilder {
    pub manager: Arc<SipManager>,
}

and it initializes everything like that:

impl SipBuilder {
    pub fn new<C, Trx, T>() -> Result<Self, Error>
    where
        C: CoreLayer + 'static,
        Trx: TransactionLayer + 'static,
        T: TransportLayer + 'static,
    {
        Ok(Self {
            manager: Arc::new_cyclic(|me| SipManager {
                core: Arc::new(C::new(me.clone())),
                transaction: Arc::new(Trx::new(me.clone())),
                transport: Arc::new(T::new(me.clone()).expect("could not start transport")),
            }),
        })
    }
}

Then each layer includes a Weak on one of its fields, since it needs somehow to notify the other layers. So for instance for Core:

pub struct Core {
    sip_manager: Weak<SipManager>,
    processor: Arc<processor::Processor>,
}

The idea is because SIP is used in various places and has many potential features in general, it should be easy to override the default (any) layer yourself, just by implementing the necessary traits and the way you want it to behave.

As you might have seen in my repo, I have implemented everything with dyn Trait strategy. I mean that works, but the problem is that I am losing the type info. And that becomes a problem actually, for instance on tests. There, when I want to test Transaction layer, that it actually sends the message correctly modified to the Core, I will inject in SipManager a SpyCore (I named it CoreSnitch), which is basically almost a double in test's terminology, and has a messages Vec in its field that I can access in tests and assert that the result.

The problem with dyn Trait, as I explained already is that I lose type information. So while I build the SipManager with it in tests, I can't access it, compiler throws an error.

And actually I understand this, it makes sense (well kinda, I am still a newcomer in Rust, but I know that it's impossible to do it that way :P).
Hence, the alternative is to use generic params instead, I hope through that I can achieve what I have in my mind. Conceptually, it's relatively simple what I am trying to achieve, but as we all know Rust with its ownership rules make it slightly more difficult than usually.

So to come back to my MVP example, I want to take it one step ahead. Ideally I would like my trait to be:

trait CoreTrait {
    fn say_hi(&self);
    fn manager(&self) -> Arc<Manager<Self>>;
    fn new<T: CoreTrait>(manager: Weak<Manager<T>>) -> Self where Self: Sized;
}

I am taking step by step, so now I am gonna skip the new function, because I have a feeling this is quite challenging to achieve and I am focusing on manager function:
I define my trait as:

trait CoreTrait {
    fn say_hi(&self);
    fn manager(&self) -> Arc<Manager<Self>>;
    //fn new<T: CoreTrait>(manager: Weak<Manager<T>>) -> Self where Self: Sized;
}

and the implementation of CoreTrait of Core type is like that:

impl<T: CoreTrait + ?Sized> CoreTrait for Core<T> {
    fn say_hi(&self) {
        println!("say hi!");
    }
    
    fn manager(&self) -> Arc<Manager<Core<T>>> {
        self.manager.upgrade().unwrap()
    }
}

however, it complaints in lines 76/77, due to return type it expects Arc<Manager<Core<T>>> (the return type) but finds Arc<Manager<T>>. But actually I am not even sure if I define the Trait method correctly, I mean to return Arc<Manager<Self>>. Maybe I should just specify to return Arc<Manager<dyn CoreTrait>> but I am afraid that I will end up in the same situation: losing data from the type.

Do you have any hint on how I should move forward?

I have a rust playground here (extended from the last one) if that helps you: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=800b2ed5d1e6a2b430250d27dcf6018f

No, it's due to the use of a bare dyn Trait. A trait object is not sized, so if you require the generic parameter to be sized, then you can't use a trait object there.


I think your life will be easier by using dyn Trait than with generics, but it's a bit unclear. Generally there are ways to let the mocked trait object sneak out information without converting it back to its original type, e.g. you could use shared memory through Arc or a message passing channel.

An entirely different approach you might consider is to spawn three tasks, one for each layer. The tasks can then talk to each other using message passing channels. It's not totally clear for me exactly how decoupled the layers are from each other, and whether this makes sense.

Hi,

Thanks for your reply. Indeed my first implementation was like that but it became extremely messy, or at least it seemed to me.

My problems using channels for communcation was:

  • channels tend to lock larger region than it should. Until you have served existing message, next message won't be server. With the new implementation I was hopping to lock things only when it is absolutely necessary
  • reasoning about the system on high load is much more difficult using channels, I think at least. I feel in high load a lot of time will actually be spent on tokio internals (since there plenty of.. channels :P)
  • apart from layers communication, when you want to introspect the system and you expect a response back, your only solution is oneshot channels, adding even more complexity/channels.

That's why I wanted to move out from channels. I was hoping that it's possible to achieve the same result without channels, but maybe it's not? You think it's impossible with generics to achieve that trait ?

It's probably not impossible.

Hi, eventually I solved it using dyn Trait and std::any:Any hack to get the concrete type back. I am using that in tests, so it should be fine right? Basically to assert my spies/doubles/mocks that they received the messages. Cause looking in the internet, I felt that anything related to Any is a hackey solution..

Thanks for the help!

Java uses type erasure (AnyObject) and type checking (type_id()instanceof) everywhere. I wouldn't call it a hack. It's simply the natural solution you'll end up with in certain cases.