Is there a fastest logger in rust?

help
1

hey, i am using rust with trading , and want to get a very fast logger in rust.
and the log is no need to show in console.

can you give me some advice?

2

Tbh, the speed of logging to a file is almost always determined by your average write speed, and not on the logging framework itself. As such, use anything that is simple - I tend to use simplelog.

3

For the loggers he's talking about the actual write is done in either another thread ,to shared memory,, or delayed until after order processing and he can afford it.

He's not talking about logging throughput but logging latency (his individual lines will prob in a cpu cache if he logged over a socket).

His big cost is usually formatting things like floats or off loading it. I wrote one that dumped the logging call stack to a shm region then patches some pointers and the receiving side basically recreate the function call on its side.

This is what he's probably referring to

1 Like
4

Depends on what you're trading and its latency requirements. Runing something some a major NASD market marker is going to have much different requirements than than some personal cloud hosts stuff against Binance.

Anywhere in that continuum you hopefully find something that dfelays any formatted and lets you log push log requests to either shm or another thread. (I prefer shm because if you crash the log is still but you might need to write a small tool to process it. Just pushing to another thread is usually acceptable as long as that thread doesn hold anything up (no buffering). Make sure it is actually pushed the data pre formatting - formatting floats is really expensive and can require memory allocations (if they are using teh Grisu3 algo which almost everybody does)..

And timestamps. You want nanos probably. It gives each log line a unique key and allows you to merge different logs on same host to reconstruct an event you want to look at.

The tracing packages are extremely heavyweight. And I have't really found a good one for similar use case. In c++ there are a couple alternatives, but I don't think Rust is there yet. Rust also seems to be based more towards throughput and safety over latency and control.

1 Like
5

jnordwick,thank you for your answer. I found that there are a lot of extremely low-latency loggers on C++, but Rust didn't have them at that time, or they didn't have long-term maintenance. I'm troubled by this

6

This may be a hit or miss suggestion, but check out https://vector.dev/ and https://www.tremor.rs/.

7

Can you elaborate on what latency constraints you have (e.g. queuing a log message needs to take less than X μs)?

Also, how you were measuring the latency in existing Rust loggers? Resolving your issue might be a simple case of using the same logging facade (e.g. tracing or log) and changing the configuration or switching to a different implementation.

8

I don't you what is your purpose but in many years ago ...
one company need to 99% uptime , and extremely use a module for almost all heavy duty application

they wrote a heavy-duty, soft real time database ( mnesia )
and mnesia internally use disk_log for persist data to disk

also they made a Logger library that production used for many years in high availability systems,

also Elixir is next language, used Logger in production,
and both language born for high availability + soft real time
project .

although both run over Beam ( like Java, Scala)

disk_log is available for Rust+Tokio

https://crates.io/crates/disk_log

Way need to disk_log ?

  • Non Blocking - optimized for tokio and
    don't block your scheduler for IO operation

  • Paging - disk_log distribute logs records between pages/files
    by total_page_size

  • Iterator - provide asynchronous function get_page then
    can iterate whole page (its safe for concurrent reading when disk_log write to it)

  • High Concurrency disk_log don't need to
    Mutex/RwLock for sharing between threads, its complete safe

  • Technically - disk_log inspired by erlang disk_log
    that is a heavily used in erlang , by mnesia database
    and Logger (Logger is production used Logger for Elixir)

9

@DanyalMh really thanks for your disk_log . i will take a look at it, exactly, i just want a non-blocking logger, and the execution of api only costs x-ns. and the formatter task will be executed in another thread. There is currently no need for async. An application will occupy a 100% cpu.

10
  1. NonBlocking
  2. Low Latency
  3. No Async

https://crates.io/crates/diskfire

11

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.