What is the best way to create a log streaming utility,

1

What is the best way to create a log streaming utility, my use case is i want to stream app logs from a remote server once a user do a deployment so that user can check the app logs at the real time for any kind of issue or error during the deployments rather than going to Splunk

2

Hello,
It's impossible to assertively say this is "best" considering that you haven't stated very much about how the streaming needs to work nor specified desirable performance criteria. Nonetheless, based on your post, I would probably start with one of the crates like flexi_logger, which has a

LogWriter trait that you may implement as needed. This (or one of the similar solutions you'll find searching of "log" or "logger" on crates.io) might give you some implementation leeway if you're as-yet undecided on the particulars.

3

Thanks plasticartsshow for the reply.
let me explain in more details what actually i needed. please suggest if there is some other better way to achieve the same thing.
here is the scenarios

  1. for example my tomcat app is running on server-1, that creates a log file at /apps/logs/catalina.out
  2. user trigger a build from Jenkins that deploy a war file on server-1
  3. here I want a utility which can read the Catalina.out in streaming mode at the server-1 and display the latest contents to the user either through CLI/webapp (Similar to what Jenkins console output shows for it jobs, but it want the logs from remote server)

current solution we have is Splunk FW on server-1, which sends logs to splunk servers and user check there. but there are some delay in this.

4

In that case, I would not follow my original suggestion – I thought you were writing the original logs with Rust. Since you already have the log files created in Java or whatnot, you'll most likely want to use one of the many file watcher crates on crates.io (like notify ) to watch the log file. When the log file changes, use one of the std rust library utils (look in std::net) to open a TCP stream or something. (Also search for "websockets" on crates.io). The notify crate also lets you "debounce" file-change events to minimize useless/repeat work.

I am not an expert, but I think that with the setup you described, you're going to run into some kind of bottleneck or delay as you described. I'm not sure if you'll get more benefit from using Rust in this fashion than you would by just implementing the log streamer in your tomcat app somehow, although it might be less overhead than the Splunk forwarder. You'll definitely want to benchmark it to tell.

5

Thanks for suggestion, will explore the options suggested by you. will update this thread once I finalized my final design and implementation choices.

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.