Hello, everyone. I have been stuck on this one for a few days. Hopefully, someone can help me with the issue I'm facing; I have inherited the project, so it could be that I'm missing some obvious bugs.
I want to immediately apologize for the long post; I wanted to add as much context as possible.
I think I have a bug similar to this one, but I'm not 100% sure.
Some context
The system I've been working on uses the Tokio runtime. Unfortunately, we have to implement sync
trait(s) because the external crate requires it, and we have no control over it.
The kicker is that the work that has to be executed in these traits is async
- I have to fetch some data using JSON RPC over IPC. Here is how I have approached this:
#[derive(Clone, Debug)]
struct FutureRunner {
runtime_handle: tokio::runtime::Handle,
}
impl FutureRunner {
/// Creates a new instance of FutureRunner
/// IMPORTANT: MUST be called from within the tokio context, otherwise will panic
fn new() -> Self {
Self {
runtime_handle: tokio::runtime::Handle::current(),
}
}
fn run<F, R>(&self, f: F) -> R
where
F: Future<Output = R>,
{
tokio::task::block_in_place(move || self.runtime_handle.block_on(f))
}
}
AFAIK this is the correct way to handle async-sync-async bridging, and most of the calls are pretty short, usually a couple of hundred microseconds, but there are a lot of them - I'm ballparking here, but we are talking range of a couple hundred up to couple thousands per millisecond.
Here is the issue I've observed:
Deadlock
I have a worker task that will do the following:
tokio::spawn
clean job which- Listens
mpsc::bounded::reciver
- On event, does cleanup work
- Listens
- Start the worker loop, which
- listens on
mpsc::bounded::reciver
channel - On event, it does some processing
- listens on
The issue is that both of these try to take the same parking_lot::Mutext
when performing some work.
This hasn't been a problem so far because this was relatively short compute work.
However, now, during cleanup work, some of these internal calls use the method described above to perform RPC over IPC calls instead of just reading from memory.
Here is the tracing
output at the point at which the app freezes (note here the lock is already held by the cleaner):
provider::remote state provider: Future:: start
provider::remote state_provider: Future:: block_in_place
provider::remote_state_provider: Future:: block on
order_input: Going to process order pool commands and take the lock
All these provider::remote state provider:*
are printed by the cleaner.
After printing out ...take the lock
, the whole program halts (even though it has enough threads at hand, both worker and blocking ones).
Note: this doesn't always happen, but it does happen often.
My guess
Here is my understanding of how Tokio works (I could be completely off base here; not an expert, not even close):
- Only one worker thread at the time can drive I/O:
- As far as I understand from the docs, even if my function is calling
block_in_place
, which callsblock_on
- all of which is happening on a blocking thread (so separate threadpool from the one in charge of runtime); still, the async tasks will be scheduled on the main runtime:- According to the docs, "Any tasks or timers that the future spawns internally will be executed at runtime."
Given the two facts above, here is what I think is happening:
- Cleaner task takes mutex lock
- It runs
block_in_place
andblock_on
code which produces some tasks that are scheduled to the Tokio runtime
- It runs
- Thread which executes worker loop at some point takes
I/O driver
lock, it reaches locked mutex, stops- Since it is stopped, it cannot drive the
I/O
anymore - The cleaner thread cannot proceed because it is blocked by
block_in_place
andblock_on
(which waits for theI/O
tasks to be finished - which is not happening) - Cleaner thread cannot release mutex
- Worker thread doesn't drive
I/O
- Deadlock
- Since it is stopped, it cannot drive the
I have "fixed" it by changing the way worker tread takes lock to try_lock_for(Duration::from_millis(10))
. Ofc. this is far from a proper fix, but before doing it properly, I wanted to understand what was happening here.
I think this wouldn't've happened had:
- The worker has been spawned using
std::thread::swpan
- The Tokio's mutex had been used since it has to be awaited
Is my analysis correct? If not, can someone please explain to me what's happening?
P.S.
The code is opensource so if need be I can share links to specific LoCs