Searching / monitoring https://docs.rs/tracing/latest/tracing/ events in real time

Problem: I've finally reached the point where printf debugging no longer works.

I have 4-5 Rust servers all logging on /app/logs.

This is currently logging events under "PrettyPrint", but I can just as easily change it to Json / OpenTelemetry formats. I don't actually care about the logging format and am happy to switch to whatever has the best tooling.

I am currently doing cat /app/logs/* | grep ... | less. I suspect there is better tooling for this.

I am interested to hear what others are using. I do need it to be something I can self host. (Not a fan of depending on third party service).

A small trick: Instead of cat you could use tail -f. This will print every line as they are added to the log files. I think it only accepts a single file as argument though so you may need to pipe the result of several tail -f commands into a single file and then tail -f this file. You could also use named pipes as log "files". You can make one using mkfifo. Of course something with rust integration would be nicer than this hack.

It supports multiple. And if your logs rotate, use -F instead of -f.

(The GNU version anyway, I didn't check any others.)