How to implement polling multiple TCP connections, some using WebSocket, when any incoming data may trigger writing to any other socket?

(Cross-post from r/rust)

Assume you want to write the following program:

  • The program opens, maintains, and, if necessary, reconnects TCP connections to different servers. These may use different protocols, some WebSocket.
  • The program listens to and accepts incoming TCP connections from a local TCP port.
  • The program runs every now and then isolated tasks that are run in separate threads.
  • Incoming data from any of these TCP connections, or the completion of a task, may require writing data to any of these TCP connections.

Assuming a *nix-like environment, writing this in C is no problem. You can run everything (except the tasks, obviously) in a single thread without anything blocking:

  • You use a non-blocking DNS library, such as c-ares.
  • All these different file descriptors used by in-progress DNS queries, incoming and outgoing TCP connections, and the local port the program is listening to, are polled using a single call to poll/select/epoll.
  • There is an extra file descriptor from pipe() that is used to interrupt this when a task completes.

How can I implement this in Rust?

  • The standard library's I/O primitives are blocking by design.
  • The existing c-ares bindings seem to require quite a lot of (platform specific) code, and apparently (?) in practise you want to launch a separate thread
  • Metal I/O (mio) offers pretty much the API I want in order to implement my event loop. However...
  • If I want to use WebSocket for some of the connections and not to reimplement the protocol or modify the existing libraries, the options are rust-websocket or ws-rs.
  • rust-websocket operates on blocking standard library primitives, so it is a no-go.
  • ws-rs actually works on top of mio, but wants to run the entire event loop itself. There is no API to grab mio's TcpStreams and TcpListeners it uses internally to use them in your own event loop.

If you want to make this even more complicated, add polling input from STDIN to the mix. Again, at least on *nix-like systems, it would be just one more file descriptor to poll.

Spawning different threads, as far as I see it, is not really an option either. If I have ws-rs waiting for data in its internal event loop, I might suddenly need to write something to its open connection because of incoming data from another connection or completion of a task. The only way to implement this I can come up with is (ab)using signals to interrupt it, and that isn't very clean... The same applies to blocking I/O and one thread per socket.

Implementing this using lots of different threads even though it shouldn't need more than one in the first place is very cumbersome. It wouldn't give me any benefit either, because what I'm trying to do involves working on a shared state, so excluding some parsing there would be only one thread doing work at a time anyway.

Any ideas?

This is the problem tokio is trying to solve. It's still very early days though.