How to design actor-based network I/O without circular dependencies?

1

Hi everyone,

I'm building a WebSocket client using the actor model with tokio, inspired by Alice Ryhl's article "Actors with Tokio". The article suggests splitting a connection into a dedicated reading task and a writing task:

Multiple actors sharing a handle
Just like you can have multiple handles per actor, you can also have multiple actors per handle. The most common example of this is when handling a connection such as a TcpStream , where you commonly spawn two tasks: one for reading and one for writing. When using this pattern, you make the reading and writing tasks as simple as you can — their only job is to do IO. The reader task will just send any messages it receives to some other task, typically another actor, and the writer task will just forward any messages it receives to the connection.

I'm trying to model this, but I've run into a design question regarding dependencies, particularly how to handle messages from the reader.

Here's a simplified sketch of my code:

// The high-level actor that handles business logic.
// It needs to send commands to the connection (e.g., via ConnHandle).
struct LogicActor {
    // Used to send commands to the connection
    conn_handle: ConnHandle,
}

// The handle for low-level connection
struct ConnHandle {
    // Used to send messages to the writer task.
    writer_tx: mpsc::Sender<Message>,
    //  How to receive messages from the reader task?
    ???: ??? 
}

// The reader task reads from the socket and needs to forward
// messages to the LogicActor.
struct ReaderActor {
    stream: SplitStream<...>,
    // How do I send messages to the LogicActor?
    // If I hold a `LogicActorHandle`, it creates a dependency cycle during setup.
    logic_handle: ???,
}

impl ConnHandle {
    // Send message to the writer task via channel.
    async fn send(message: Message) { ... }

    // How to design the api for receiving messages from the reader task?
    ???
}

// Main setup logic
fn setup() {
    // 1. Create the LogicActor, get its handle.
    let logic_handle = ...;

    // 2. Create the connection actors.
    // The ReaderActor needs the `logic_handle` to be constructed.
    let reader = ReaderActor::new(stream, logic_handle);
    
    // 3. But what if the LogicActor also needs the `ConnHandle`
    //    to send messages? Now we have a circular dependency for construction.
}

My questions are:

  1. The article seems to suggest that the reader and writer actors can share a common handle. However, their communication patterns are different: the writer receives messages to send, while the reader produces messages for others. How is a shared ConnHandle typically designed for this?

  2. This leads to my main problem: What is the idiomatic way for a LogicActor to receive data from the ReaderActor? If the ReaderActor holds a handle to the LogicActor, it creates a circular dependency during construction.

  3. How can I resolve this construction-time dependency cleanly, especially when the LogicActor also needs the ConnHandle to send data out?

2

I think you need to separate the actors and handles, so that every actor only owns handles (not actors owning actors).
The handles structs should have methods for what you can do with them: ConnHandle can have send_to_writer(&mut self, message: Message).
Then the only actors that own ConnHandle are the ones that need to send messages to the writer.

For a bidirectional setup, it makes more sense to have two handles structs, since the writer shouldn't be able to send_to_writer.

A list of the methods you're planning to have on each handle struct can help guide the overall design.

3

Apologies, rereading your questions and that part of the article, what I think they mean is:

One handle type, and (privately) behind it are two concurrent tasks running. One to handle all the reads, and one to handle all the writes. Basically, they conceptually "split" the handle to execute their task.
But from the point of view of the rest of the program, it's still one handle. The fact that it can do reads and writes concurrently is an implementation detail.

More concretely: when ReaderActor needs to return a response to the caller, it should use the oneshot channel that was included in the request.
That way, the reader doesn't know who sent the request, it just supplies the response data (or cancels if the requestor dropped their half of the oneshot channel)

4

Thanks for the comment!

Actually, the WebSocket service exposes two types of APIs: request/response (which includes an "id" field for matching) and data stream subscriptions.

And my intention for ReaderActor is actually a bit different. I’d like to keep it focused purely on low-level WebSocket protocol handling (e.g., PING, PONG, CLOSE), without involving higher-level concerns like request/response matching or messages classification.

So with that separation of concerns in mind, I’d like to return to my original question:

How should the ConnHandle/ReaderActor expose its API in a way that allows higher-level logic (e.g., a LogicActor) to receive and process messages appropriately?

In other words, what's a clean and idiomatic way for the reader to hand off parsed WebSocket messages to a higher-level component?