Time based profiling


I am working on multi threaded app using Tokio and I occasionally witness a lag in display. When the app was single thread, I just did perf flamegraph and looked "which function eats cycles" and optimised there, worked like a charm multiple times. But now the app is more complicated, it uses rust-analyzer in background and calls to it dominate perf profile (which btw is interesting, because I didn't realize perf is so smart to look into subprocesses).

Is there a way to see "which functions take the most time", not cpu cycles? I expect that something is waiting for IO with other threads, causing the slowdown but I don't know how to display it this way. Also, I would like to scope my investigation to my code only, I won't optimise rust-analyzer and I don't care what it does, I just want to know "which function in my code waits for it" so I can put it in a different thread than UX.

Here's the app Andrzej J. Skalski / bernardo · GitLab , I use "cargo flamegraph" to build perf.data and then hotspot to view it. I am not sure do I need to change the flamegraph command or perf.data is good and I just need to query it in some smart way.

Can someone help me out?

1 Like

If you annotate your code with tracing, then the resulting tracing spans should reveal this information. However, this requires manually annotating your code with tracing, so its not a super easy solution.

A possible approach is loading your profile into hotspot and either using an inclusion-filter on a critical single thread (if your workload is structured that way) or several exclusion-filters on the irrelevant background tasks.