Suggest a crate: Inter-process 'multicast'?

#1

I use UDP multicast to have multiple consumer processes on a single host receive copies of the data from a producer on the same host – ‘tens’ of MB per second / ‘thousands’ of packets per second. This setup has some useful properties that I want:

  1. Broadcast messaging model
  2. Bounded queues
  3. Slow consumer looses data; does not cause problems for others
  4. Process isolation for each component

However, trying to make it more efficient would be good sport!

Is there are Rust crate that already supports an IPC model like this without sending everything via the kernel and copying all the data for each consumer process?

#2

Be careful what you wish for. “Avoiding copying” sounds nice, but the alternative in your circumstances is generally some kind of garbage-collection scheme and more cross-process synchronisation.

This might not be so bad, if the garbage collection scheme is just “keep the last N entries” in a circular buffer or similar. It should be quite feasible to do in a shared-memory segment, for example. I don’t have a specific recommendation for something that does exactly this, but this might give you some extra search terms.

I would look at the rust bindings to ZeroMQ as a good place to start, for more general solutions.

#3

It sounds like UDP multicast is the least bad solution here.

You could use pipes to let the kernel skip all the networking work and just copy data to an internal buffer the child process can read from, but that would require maintaining one pipe for each child process and may not end up being faster (n writes required for pipes, vs one UDP send).

There are also ways multiple processes on a Linux machine can share memory (e.g. mmap), but that sounds like a great way to introduce data races and soundness/safety issues (as @dcarosone alludes to).

Are there any other low-hanging fruit you could use for improving throughput? For example, you may be able to minimise the amount of data transmitted, use compression or a more efficient format, or write everything to disk so clients can process the data over time.