So ... I Wrote a Light Weight Low Latency Logger

1

Hey everyone,

I spent some of my free time writing a light weight logger if anyone is interested in checking it out! I mainly wrote this as I did find in my low latency app logging was causing the majority of my bottleneck. Link below:
https://crates.io/crates/inqjet

The architecture follows standard async logging, but optimized to reduce latency from the producer threads and achieve very consistent tails. Below are some HDR histogram results from empirical benchmarks in the repo:

=== Tracing Performance Profile ===
Count: 100000
Min: 0.46μs
Mean: 1.38μs
Max: 120.83μs
StdDev: 0.99μs

Percentiles:
50th: 1.20μs
90th: 2.05μs
95th: 2.21μs
99th: 3.29μs
99.9th: 13.86μs
99.99th: 26.66μs

=== InqJet Performance Profile ===
Count: 100000
Min: 0.20μs
Mean: 0.55μs
Max: 44.58μs
StdDev: 0.45μs

Percentiles:
50th: 0.45μs
90th: 0.91μs
95th: 1.01μs
99th: 1.63μs
99.9th: 5.02μs
99.99th: 13.69μs

It by no means is as feature full as tracing, so if you rely on specific features in that crate, this might not be for you. I however did add some configurations so you can tune the logger based on your expected logging behavior.

The above benchmarks were tuned to achieve as low latency as possible in the benchmark. For realistic workflows, you can easily trade off cpu usage for latency, depending on your preferences. For the lowest latency, there is a busy-spin mode, but I would only suggest that if you are okay burning a cpu core in your background appender thread :slight_smile: .

5 Likes
2

This is interesting, but there is no latency guarantee. What would it take for a logging framework that supports hard realtime constraints?

Another related question: have you compared with defmt, which is used in embedded and offloads the formatting to the host computer. Moving the formatting to a separate thread would further reduce the latency in your case I would assume.

3

Nice!

4

Ah I was not aware of that crate! Yes, the main bottleneck at this point I believe would be the string formatting. I had attempted it along the way before I published the crate to defer as much as possible to the background thread which had some wins.

I wanted to figure out a better solution than what I had built, so for now I just used the string pool, since I found that a lot of the latency was coming from the allocator or the sys calls when the logger is cold.

5

Recently I wrote a very simple logger, with a log rolling feature. If your logger supports the feature, I am ready to switch to yours.

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.