I'm trying to troubleshoot an application that seems to be leaking some async tasks.
I'm using tokio::runtime::RuntimeMetrics and I'm monitoring alive_tasks.
I'm testing my application by sending some load, then going idle for like 30s, in an endless test loop. Over time, I see how alive_tasks keeps growing.
After trying to add logs and also ensure JoinHandles are at least aborted on drop, I still see alive_tasks growing (and base memory usage slowly growing as well).
So I tried to use tokio-console.
I'm able to connect and I see information about my tasks, but I'm very confused because the output seems to be updating every second with new information, even when the test is not running which means my application should be pretty much idle.
There is some minimal housekeeping work triggering in my application but the information about the tasks that I see for 1s on the screen refers to code locations that are executed during the test, and the test is not running.
Is it possible that tokio-console is somehow buffering information and it's not really real-time and is showing me delayed information?
After stopping the test and keeping the application and tokio-console running, tokio-console information seems to become "stable" and not changing every 1s. But it does take several minutes after I stopped the test, which again sounds as if there is a backlog of information tokio-console still needs to process.
Does anybody have a good explanation for this? or any suggestions on how to troubleshoot this further?
not that I can see (but bear in mind this is the first time I use tokio-console so I may well be missing it.
I've attached a screenshot.
The UI keeps updating every 1s well after my test client has finished and my application logs are "quiet", so again it looks as if the UI refreshes are "behind" realtime.
So, tokio-console does buffer information to some extent, but the buffer has a maximum size above which events are dropped instead of falling too far behind. When that happens, there is a message at the top of the window like this:
It doesn't look like that's happening in your case.
So yes there is buffering and it can be a bit behind and then catch up. But it would be surprising to me that it could get >1 minute behind without showing this error.
Are you sure that the tasks aren't actually running? The might still be running even though the test has finished if the test did not wait for them to exit.
thanks. I'll keep checking as I'm likely missing something.
This is a grpc/rest application and I'm the only one sending grpc requests to it. Running on my laptop.
Those grpc requests are handled by tonic and also spawn some application-specific tasks.
The test client I'm using is a grpc client that gets proper success responses for all grpc requests, so there shouldn't be any other tasks being started other than the ones which started from grpc requests.
I'm also using tokio::runtime::RuntimeMetrics to expose tokio metrics through prometheus metrics. It's not exact math, but when I query metrics while the test is running I do see alive_tasks going up. When my test client is done and the application runs idle, the alive_tasks metrics stay flat on the same baseline numbers they had before I started the test client.
The application-specific tasks for example handle grpc streams.
My test client is short-lived; makes a grpc request that gets a stream, and disconnects. The stream handling task is (I believe) finished, and the alive_tasks metrics seem to confirm that.
After finishing the test client, several seconds and perhaps even 1min after, on the tokio console I see "flickering" tasks for 1s on and off, pointing to code that handles grpc streams; tasks that should be already finished many seconds ago (and again alive_tasks seems to confirm this).
anyway, thanks again and if anyone has any suggestion on how to analyse this further please let me know.
