Request for feedback: how to clean up actors

help
1

I'm building an actor library and have ran into a sticking point. For context, the library is powered by macros, so I have a lot of leeway in terms of generating or modifying the implementation for a given actor on top of the code provided by the user. However, two things I want to support in the design are:

  1. I want you to be able to implement traits on your actors and upcast them into trait objects (or equivalent) so the API to create an actor of type Actor hands out Arc<Actor>. I need to use Arc specifically here because being able to upcast "through" a type, i.e. to produce a Arc<dyn Trait>, is governed by CoerceUnsized which is unstable. Except for the issue I'm about to explain, the lifecycle of the actor is supposed to be tied to the Arc values as you'd expect - the actor persists, receiving any messages sent to it, until all Arc values pointing to it are dropped, at which point it processes any remaining messages and drops its internal state. (It is also possible for the actor to stop for a couple of reasons without all of the Arcs dropping, but I know how to handle that.) Conceptually, the actual value behind the Arc is a queue-sender connected to the actor's message loop.
  2. when the actor's message handler runs, there is an API available to it that gives out a copy of the actor's own Arc<Actor>. This is provided because various patterns used for actors and async queues involve passing your own address as a destination/receiver, and forcing a client that needs to do that to manually make it explicitly part of the actor's state is unnecessarily awkward. Naively, we might do this by storing a Weak internally so as to not disturb the actor's lifetime and then upgrade it as needed.

However, now there is a problem: the Arc values are owned on other tasks (in the async sense) by whatever client code wants access to the actor, whether other actors or other arbitrary parts of the system. OTOH, the user's message handlers run on the actor's message loop, which is defined as its own separate task. This means that there is a scenario where the actor's last Arc might drop while there are still messages waiting to be processed. The handler for that message might then call the API for a copy of its own address and... the Weak fails to upgrade, so we can't return the contracted Arc.

Potentially, the API could instead return a Weak (or Option<Arc<_>>) and let the user handle the potential failure to upgrade, but I don't like this option because I expect that in almost all cases, asking for the address and not being able to get it can only be handled by the actor self-destructing by panic, and forcing every client to deal with that is conceptually messy and leads to a boilerplate else { unimplemented!() }. Additionally, it seems to cut against the entire point of the actor model managing state changes by tying them to messages being passed - instead, we effectively have the actor (irrevocably) changing from a "full" to "walking dead" state, spontaneously and untethered from any information the actor has access to.

Instead, I have come up with various solutions that allow the API to return Arc and always succeed even if all external handles have disappeared. However, they all have tradeoffs:

  1. Store the strong Arc inside the event loop, so we can always hand it out. This creates a situation where if all the external Arcs disappear when we don't have any messages pending, the event loop hangs forever because it is waiting on a sender that remains alive because the event loop itself is holding one of its the Arcs, and (because it is waiting) it does not get an opportunity to realise that its internal Arc is the only one in existence and it should shut down. To fix the hang, I put a timer inside the event loop, so the actor artificially "gets a message" even with no other senders, at which point the check can happen and close everything down. While this solves the hang, it also means dead actors do not release their memory when their last Arc drops, but instead potentially as late as the next time their timer fires. While I can offer the user control over how often the timer goes off, this would still mean idle actors create some non-zero CPU load simply checking if they need to stay alive, although I expect it'd only be a few operations per actor per wakeup in practice.
  2. Store the strong Arc as above, but instead of each individual actor having a timer, have a single global "supervisor" actor that unblocks loops by sending all other actors occasional messages even if no other senders exist. This means the CPU load for collecting garbage no longer scales with the number of actors in existence, but is nonetheless still not zero. This is the option I've investigated least, and in particular I have not dug into issues like how such a supervisor should work so as to both 1) not potentially flood the mailbox of actors that are still alive and/or still receiving messages, 2) not allow a dead actor to hang around indefinitely. However, I think this is least intrusive as far as user code goes, because subscribing to supervisor can be done entirely behind the scenes by the macros that produce the actor machinery in the first place and any knobs it has to tweak are less important. That said, I do not like this option because it creates a point of centralisation and bottleneck, various overheads, and the rest of the library has avoided having global structures of any kind up until now.
  3. Store a Weak and allow the Arc to drop, but recover the underlying sender and store it within the event loop. This means in the original problem scenario where the message handler asks for its Arc, and the original Arc has dropped, the underlying sender is still available, meaning we can allocate the sender into a new Arc, and hand that out. This seems to maintain the API, at least as far as typing and relational logic goes - the handler gets an Arc, and that Arc points to the sender corresponding to the same message queue the handler is processing. However, it has a potentially surprising downside: from the POV of a component holding only a weak reference to an actor, it is possible for the "same" actor to reappear with a different address at a later point. (This can only happen if all of the strong Arcs drop and, afterwards, the message handler hands out a copy of the address. So long as anyone anywhere holds a strong Arc, the address is stable.) I would appreciate any comments on the impact this might have in practice, because I cannot think of any use cases where we would hold a Weak [1] but it is somehow semantically important that 1) even though ptr_eq returns false, 2) the reallocated "same" actor can't be treated as a completely new object. That said, the upsides of this method are appealing: there are no knobs to tweak, no CPU load us created by idle actors, and it Just Works™ if you don't care about using the address of the actor as an identifier. (AFAICT using the Arc implementations of Eq or Hash do not involve the underlying address so any comparison using those would be unaffected.)

I've come here to ask for feedback because I'm not sure which method to choose. If you were using this library in a project, which of these do you think offers the most in terms of making it easier to reason about actor interactions and how your software will behave? Which is most understandable, and/or easiest to parse in terms of performance characteristics?

Please feel free to mention any other solution that I might have missed too.

  1. which by nature might be dead at any time ↩︎

2

I'd suggest the semantics you want are basically "free if there are no external handles or messages in the queue for this actor", so at least semantically each message includes an Arc for the destination.

In terms of making that efficient I'm not sure, since it depends on your internals. There's a few options, but it's pretty tricky to make race free unless you're in control of the code for both the message queue and the reference count.

3

So depending what you mean, I don't control the reference counting directly because I need to be using Arc. (If it weren't for the CoerceUnsized constraint I could use my own smart pointer that could run some code when all-but-one references disappear.) However, I can manipulate those references freely, store them in the actor metadata, message loop, etc. Manipulating the strong/weak counts directly is also an option but I haven't had any ideas about how it'd help this problem.

As for the message queue, I'm aiming to be generic over the queue type (although adding requirements to it would be worth it to make this work well) but I have full control over what values go through that queue.

Directly adding an Arc to every message payload is viable, and sounds race-free to me, although it would mean a lot more contention on the ref count. I'm not sure if that's what you meant by "efficient" or if you had something else in mind.

4

Yeah pretty much. I don't know how well you could reduce the contention on the arc without racing or a lock on the message queue, which kind of defeats the point, but I've not done a proper analysis.