What does it mean "tasks have lost their waker"?

When I use tokio-console to observe tasks on my server, I see plenty of warnings about tasks that have lost their waker.

I'm not implementing any Futures myself, and not working with Wakers directly. In my code the warnings point to Handle.spawn() calls that I use to move futures from Actix-Web's runtime to a dedicated tokio runtime.

Is this an expected behavior? Does spawn() require some special care to avoid losing wakers? Or is this a tokio bug?

╭Warnings──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│⚠ 43 tasks have lost their waker                                                                                                                                                                                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭Tasks (9195) ▶ Running (1) ⏸ Idle (72)────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│   Warn▿ ID   State  Name  Total      Busy       Idle       Polls  Target                Location                                                                                          Fields                                                                             │
│   ⚠ 1   500  ⏸            1387.9044s   7.7468ms 1387.8966s 41     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.23/src/common/exec.rs:49:21  kind=task task.id=4624                                                             │
│   ⚠ 1    46  ⏸            1434.7280s   13.1538s 1421.5742s 45     tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=42                                                               │
│   ⚠ 1   5429 ⏸             658.7376s    3.6499s  655.0877s 68     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=27210                                                           │
│   ⚠ 1   9497 ⏸             112.8113s    2.2243s  110.5870s 43     tokio::task           server/src/main.rs:1044:46                                                                        kind=task task.id=47029                                                            │
│   ⚠ 1    282 ⏸            1418.0028s   1.8127ms 1418.0009s 19     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.23/src/common/exec.rs:49:21  kind=task task.id=2451                                                             │
│   ⚠ 1    164 ⏸            1421.7893s   22.0456s 1399.7437s 10     tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=782                                                              │
│   ⚠ 1   3144 ⏸            1018.6552s  33.4192ms 1018.6218s 22     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=17936                                                           │
│   ⚠ 1   5527 ⏸             627.2128s    1.0006s  626.2122s 37     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=27940                                                           │
│   ⚠ 1     24 ⏸            1434.7994s    1.0185s 1433.7809s 3656   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=26                                                           │
│   ⚠ 1   8114 ⏸             292.2537s    1.7847s  290.4690s 325    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=42591                                                           │
│   ⚠ 1   8992 ⏸             189.8128s    5.2512s  184.5616s 372    tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=45444                                                            │
│   ⚠ 1      5 ⏸            1445.0162s   12.5179s 1432.4983s 43074  tokio::task           server/src/main.rs:118:21                                                                         kind=task task.id=12                                                               │
│   ⚠ 1      6 ⏸            1445.0171s    2.3804s 1442.6367s 2225   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=11                                                           │
│   ⚠ 1      7 ⏸            1445.0192s  87.6610µs 1445.0191s 2      tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=2                                                            │
│   ⚠ 1     15 ⏸            1434.7926s   2.9970ms 1434.7896s 2      tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=37                                                              │
│   ⚠ 1     19 ⏸            1434.7994s   2.9487ms 1434.7964s 2      tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=27                                                              │
│   ⚠ 1   5540 ⏸             627.0149s  17.8065ms  626.9971s 110    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.23/src/common/exec.rs:49:21  kind=task task.id=27986                                                            │
│   ⚠ 1     22 ⏸            1434.7288s    3.8506s 1430.8782s 25     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=41                                                              │
│   ⚠ 1     33 ⏸            1434.8045s   3.0358ms 1434.8015s 2      tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=22                                                              │
│   ⚠ 1   4315 ⏸             822.8296s    2.7743s  820.0553s 14     tokio::task           server/src/main.rs:1044:46                                                                        kind=task task.id=22680                                                            │
│   ⚠ 1     30 ⏸            1434.7927s    1.0939s 1433.6988s 3664   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=36                                                           │
│   ⚠ 1     36 ⏸            1434.8046s    1.0542s 1433.7503s 3690   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=21                                                           │
│   ⚠ 1   7748 ⏸             348.7509s    7.3059s  341.4450s 210    tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=40493                                                            │
│   ⚠ 1     28 ⏸            1434.7926s 328.3037ms 1434.4643s 3124   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=38                                                              │
│   ⚠ 1   7516 ⏸             390.9141s    6.0391s  384.8750s 139    tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=38992                                                            │
│   ⚠ 1    434 ⏸            1390.7964s    3.8085s 1386.9879s 520    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=3830                                                            │
│   ⚠ 1    160 ⏸            1421.7908s    5.2053s 1416.5855s 912    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=781                                                             │
│   ⚠ 1   1227 ⏸            1248.7096s   15.7769s 1232.9327s 2076   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=10515                                                           │
│   ⚠ 1   2102 ⏸            1197.8422s   13.2051s 1184.6371s 16     tokio::task           server/src/main.rs:1328:8                                                                         kind=task task.id=13430                                                            │
│   ⚠ 1   7749 ⏸             348.7518s    3.3869s  345.3649s 36     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=40492                                                           │
│   ⚠ 1   5424 ⏸             658.7372s    4.4584s  654.2788s 7      tokio::task           server/src/main.rs:1044:46                                                                        kind=task task.id=27211                                                            │
│   ⚠ 1     68 ⏸            1432.4212s   1.7776ms 1432.4194s 19     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.23/src/common/exec.rs:49:21  kind=task task.id=63                                                               │
│>> ⚠ 1     23 ⏸            1434.8045s 375.4976ms 1434.4290s 3125   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=23                                                              │
│   ⚠ 1     21 ⏸            1434.7961s   3.1808ms 1434.7929s 2      tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=32                                                              │
│   ⚠ 1     34 ⏸            1434.7961s 307.2397ms 1434.4888s 3124   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=33                                                              │
│   ⚠ 1     35 ⏸            1434.7962s 964.6510ms 1433.8315s 3662   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.7.0/src/runtime.rs:80:20     kind=block_on task.id=31                                                           │
│   ⚠ 1   7515 ⏸             390.9146s    3.4207s  387.4939s 652    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=38991                                                           │
│   ⚠ 1   4306 ⏸             822.8316s    2.6076s  820.2240s 77     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=22679                                                           │
│   ⚠ 1   2100 ⏸            1197.8431s 921.3859ms 1196.9217s 16     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=13429                                                           │
│   ⚠ 1   8991 ⏸             189.8135s    2.2319s  187.5815s 673    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=45443                                                           │
│   ⚠ 1   3145 ⏸            1018.5131s  86.8039ms 1018.4263s 453    tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.23/src/common/exec.rs:49:21  kind=task task.id=17945                                                            │
│   ⚠ 1     20 ⏸            1434.7994s 338.1718ms 1434.4612s 3124   tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=28                                                              │
│   ⚠ 1   9496 ⏸             112.8117s    1.1366s  111.6750s 45     tokio::task           /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/local.rs:317:32   kind=local task.id=47028
1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.