Recursive log call

help
1

I'm trying to implement simultaneous logging to stderr and MQTT via the paho_mqtt library. I'm using simplelog::CombinedLogger with a TermLogger and a WriteLogger.

I have a struct that implements io::Write to handle the MQTT side of things. It buffers write() calls until it gets a newline then publishes the buffer to a predefined MQTT topic. It works fine when I have INFO or lower logging but as soon as I enable DEBUG the program locks up.

It looks like paho has a debug!() call in the publish function which tries to recursively write to my WriteLogger. WriteLogger guards calls to log() with a mutex so I get stuck trying to recursively grab the mutex.

I think I may need to create a custom logger interface instead of using WriteLogger and do some special filtering on paho related messages. Before I do this I figured I'd ask if there are better solutions. Perhaps a different logging library?

2

Consider writing only to stdout, and using a separate system to process logs for re-publishing to MQTT (or other log aggregators).

This is Factor 11 of The Twelve-Factor App.

3

Having written things that implement this pattern for complex logging, I'm not convinced it's a universal win. The twelve-factor pattern is a specific platform design document, and while it does have widespread applicability (especially since it influenced the design of later platforms), it's not universal. Apps doing their own logging have their place.

Having a log filter that both records stdout+err to a file and emits them to an MQTT broker isn't easy to get right - possibly harder to get right than doing it in the source program.

4

Not an option. MQTT logging is the primary method and console logs are used for debugging. System is running on a fairly tight embedded system without syslogd or similar. Attempting to take the mqtt logs and output them to the console for debugging is fine but my probably still exists (i.e. recursive logging to mqtt)
Besides, the 12 factor app methodology is really only applicable to SaaS situations (per the intro and scope)

5

I think your conclusion is supporting evidence for just not doing it in the source program in the first place. Whatever complexity makes it not easy is better encapsulated elsewhere. The application isn't a log broker itself, but giving it this functionality makes it a business logic concern that should not exist.

Apart from that, I agree that in principle programs that do their own complex logging have a place. (See OP's response WRT MQTT being the primary log destination.)

The good news is that we are beginning to unravel the actual question. "How to log in an embedded environment" is an important component of that question. And I might add, it's a component missing from the original question as posed. My previous answer does not make sense with this additional context, but then I would not have proposed it had I been given the context in the first place.

simplelogger::WriteLogger is not meant to use Write implementations that themselves use the logger. It's meant more for actual stream implementations, like File or TcpStream. AFAICT, paho_mqtt doesn't provide any streams that implement Write, but it stands to reason you might want to use it to create one.

You may be able to break the recursion by creating an app-specific logger instead of installing it globally. But then you miss all logs that use the global logger, including logs from the mqtt crate. Not that you will want them, but presumably they exist for a reason, i.e., debugging.

Filtering might be another option. Again at the expense of logs from the mqtt crate.

I think there is still something to be learned from the 12 factor app methodology. Namely that there is no good reason to throw away logs from the MQTT implementation simply because you want to use it for shipping logs. For this to work, you need to break the coupling. 12 factor imposes a constraint that logs should be treated as streams instead of destinations and formats. This nicely solves the problem because a pure stream doesn't need to log anything - it's just a sequence of data, not behavior over some data.

Concretely, a stream is something like a channel. Implying that writing to the stream is just the operation of sending to a channel, and some other process is responsible for consuming from the stream and sending it elsewhere, using MQTT for instance. The MQTT library can send its own logs to the stream. These are not recursive calls; The consumer will just pick up the new log on the next iteration. It means that you still need some filtering to avoid log amplification filling the channel. But otherwise it helps break the recursion.

Anyway, looking over paho_mqtt, I'm surprised you have the standard library in your embedded system. Does this mean you also have threads? A stream consumer in its own thread can install a no-op thread-local logger and not deal with any logs generated on its own thread. Do you have processes? Process isolation is even better! Is it possible to use a different mqtt protocol library with the standard library TcpStream? Etc, etc.