CPU intensive Tokio tasks end up on the same thread

gak · March 26, 2021, 5:18am

Hi!

I'm running a CPU intensive function over multiple tokio tasks while sending stats via a mpsc into another task to print out stats. Initially all the CPU threads are maxed out as expected.

The problem I saw was the total speed drop over time (and CPU usage drop), and suspected that all the tasks ended up running on one thread. After investigating that seems to be the case. Figured it out by adding an occasional print of thread::current().id() and after 15 or so minutes they all end up being the same thread!

How can I keep them on different threads? Isn't this a strange thing to happen?

Small note: Removing the yield_now() doesn't make a difference.

tokio = { version = "1.4.0", features = ["full", "rt-multi-thread"] }

github.com

feeless/feeless/blob/main/src/vanity.rs#L102


    let cpus = num_cpus::get();
    let attempts = Arc::new(RwLock::new(0usize));
    let tasks = self.tasks.unwrap_or(cpus);
    let (tx, rx) = tokio::sync::mpsc::channel::<SecretResult>(100);
    info!("Starting {} vanity tasks", tasks);
    for _ in 0..tasks {
        let v = self.clone();
        let tx_ = tx.clone();
        let counter_ = attempts.clone();
        tokio::spawn(async move {
            v.single_threaded_worker(tx_, counter_).await;
        });
    }
    Ok((rx, attempts))
}

async fn single_threaded_worker(&self, tx: Sender<SecretResult>, counter: Arc<RwLock<usize>>) {
    while !tx.is_closed() {
        for _ in 0..self.check_count {
            if let Some(result) = self.single_attempt() {
                if let Err(_) = tx.send(result).await {

Thanks for any help

Michael-F-Bryan · March 26, 2021, 8:22am

You should probably be using tokio::task::spawn_blocking() here. The tokio::spawn() function expects you to spawn a "normal" IO bound future, whereas spawn_blocking() will run the task on a background thread.

gak · March 26, 2021, 8:36am

Thanks for the response! IIUC, spawn_blocking is for when you don't run async. I still want to send messages through a channel.

Besides, I can just use threads directly or something like rayon. I'm just curious why an outcome like this is happening though, and if it's possible to fix? It starts perfectly fine, then moves all those tasks to a single thread for some reason!

alice · March 26, 2021, 10:58pm

I have written a blog post that explains how you should handle CPU intensive code in async: Async: What is blocking?

Tokio's scheduler is optimized for tasks that spend most of their time being idle, and it may make incorrect decisions if they are very CPU-intensive.

Note: You can still use your channel in non-async code through blocking_recv or blocking_send. However, be aware that listening on channels in the rayon thread pool is a bad idea for the same reason as why the blog post earlier recommends dedicated threads for things that run forever.

alice · March 26, 2021, 8:45am

Note that using the std yield_now like that doesn't make much sense. You need to use the one in Tokio if you want to actually yield to the executor.

gak · March 26, 2021, 10:58pm

Amazing, thank you alice.