Hey everyone, I wrote some compute heavy code in rust that:
in parallel (by spawning and waiting for a semaphore permit)
gets some data from a database, .awaits it
computes on the data in a par_iter (inside a spawn_blocking)
Maybe I could have two concurrent tasks, one is downloading and awaiting, which sending via a channel into the rayon-based compute task, would that be better? (I'll experiment at some point and edit this post)
But the crux of this post is that I noticed htop displays many red bars when this code is being executed, and i couldn't find any better explanation of red bars as "kernel threads"
I figure many blue/green bars > many red bars, and I might be missing out here on major performance wins.
Could anyone explain what Red bars may indicate, and/or your top suspicion for the cause? (Am I making many sys-calls, or allocations, etc?)
How much data are you getting at a time? If it is only a single row, the overhead of getting the row from the database and passing it to the threadpool may be much bigger than the actual work.
Oh it is one "row" at a time, but it's actually some binary data stored in Elasticsearch (not a traditional db as such).
But in my example where I see the high red bar activity, i only do this 56 times, i have this sempahore strategy, because in the real world it could potentially be many thousands. (and I don't want it all sitting in RAM)
The work that is done on this data is definitely not the part that takes the longest, but if
the overhead of getting the row from the database and passing it to the threadpool may be much bigger than the actual work
were true, out of curiosity, what would you have suggested? Getting it in chunks at a time? (because I can't get it all at once)
Yes, batching many small tasks together often reduces overhead, improving throughput.
I see. In that case it is probably not the issue. If you are on linux you could try perf for profiling. It is capable of showing not just where you spent time in userspace, but also in the kernel when perf_event_paranoid is 1 or lower (or you run it as root).