I just went a bit to the drawing board, and I think that messages actually do not need to be fixed-size. All they really need is to be serializable, and to know an upper bound of their size in bytes in advance...
...however, the support data structures and algorithms for variable-sized messages that I came up with are nasty, so if I end up implementing this, the first prototype will definitely use fixed-size messages over a standard MPSC bounded queue implementation
Ideas for variable-sized messages
Basic protocol for variable-sized push is as follows:
- Determine how much space you need
- Allocate a block at least this large inside a fixed-sized storage block
- Fill it up using your favorite message serialization protocol
- Send coordinates of the message's storage slice over a classic MPSC queue
- If allocation or send fails, increment buffer overrun counter
Once this nasty part is taken care of, popping is straightforward:
- Pop message slice from the MPSC queue
- Read data from the slice and deserialize it (can actually use memory allocation here, we're on the logging thread's side)
- Liberate storage block after use
- Notify caller of buffer overruns, if any, and reset the counter
The tricky bit here is of course push step #2, which effectively amounts to implementing a concurrent memory allocator with liberation support. It doesn't need to be the fanciest allocator in the world, a bitmap allocator with a small block size would do just fine for this undemanding task. But it's still quite a piece of work.
Further, while sketching this, I also reached the conclusions that my desired semantics of dropping old messages to make room for new ones would be a royal pain to implement, especially in a lock-free way and for variable-sized messages. I don't care that much about it since dropping messages should never happen, so I'll give up on that requirement.