Any real-time safe logger around?

I just went a bit to the drawing board, and I think that messages actually do not need to be fixed-size. All they really need is to be serializable, and to know an upper bound of their size in bytes in advance...

...however, the support data structures and algorithms for variable-sized messages that I came up with are nasty, so if I end up implementing this, the first prototype will definitely use fixed-size messages over a standard MPSC bounded queue implementation :wink:

Ideas for variable-sized messages

Basic protocol for variable-sized push is as follows:

  1. Determine how much space you need
  2. Allocate a block at least this large inside a fixed-sized storage block
  3. Fill it up using your favorite message serialization protocol
  4. Send coordinates of the message's storage slice over a classic MPSC queue
  5. If allocation or send fails, increment buffer overrun counter

Once this nasty part is taken care of, popping is straightforward:

  1. Pop message slice from the MPSC queue
  2. Read data from the slice and deserialize it (can actually use memory allocation here, we're on the logging thread's side)
  3. Liberate storage block after use
  4. Notify caller of buffer overruns, if any, and reset the counter

The tricky bit here is of course push step #2, which effectively amounts to implementing a concurrent memory allocator with liberation support. It doesn't need to be the fanciest allocator in the world, a bitmap allocator with a small block size would do just fine for this undemanding task. But it's still quite a piece of work.

Further, while sketching this, I also reached the conclusions that my desired semantics of dropping old messages to make room for new ones would be a royal pain to implement, especially in a lock-free way and for variable-sized messages. I don't care that much about it since dropping messages should never happen, so I'll give up on that requirement.

2 Likes