Is there a way to know where a mutex was poisoned from?

1

The title says it all. I have a "database" mutex for sharing a database connection between threads and at some point, it's poisoned. I have no idea why / where does it get poisoned from because it gets poisoned randomly at some point while it shouldn't even be used. Ideally, I would like to have that rust backtrace thingy displayed, for now I just have this

image
image692×192 35.4 KB

2

A Rust mutex only gets poisoned when a thread panics while holding a lock. This panic will print a message. The messages you show don't come seem to come from rust though. What is the code acquiring the mutex?

3

No panic message were printed, locks happen everywhere in the code, but the code that checks for locks is this one :

    std::thread::spawn(|| {
        let duration = std::time::Duration::from_secs(60);
        loop {
            std::thread::sleep(duration);
            if database::DATABASE.is_poisoned() {
                println!("Database mutex poisoned ({}).", database::DATABASE.try_lock().err().unwrap());
            }
            else if database::DATABASE.try_lock().is_ok() {
                println!("Database mutex free.");
            }
            else {
                println!("Database mutex currently locked");
            }
        }
    });
4

It's certainly caused by a panic in code that uses the mutex.

It's possible that you're not seeing panic printed, because:

  • If panic happens on a thread, it's returned as an error from the thread's JoinHandle. Make sure you handle errors from thread.join() if you use threads.

  • There may be code that uses catch_unwind. 3rd party libraries that take callbacks often use it to catch panics from callbacks and report them in some other way.

You can abort on panics to make the failure more evident.

Alternatively, take a closer look at your database code that uses the mutex, and replace all .unwrap()/.expect() calls with graceful error handling. If you have array access vec[x], that can panic too if it's out of bounds. Review that or use .get(x).ok_or("oops")? instead.

1 Like
5

The thing is that those thread that use the muted aren't made by me, they're made by frameworks like rocket or serenity. My rocket logs are set to critical, maybe that hides the panic messages ?

6

I haven't used Rocket, but it's very likely that a server framework catches panics from each request, so that a single request can't take down the whole server.

1 Like
7

Ok, I'll take a look at that

8

Incidentally, I have only used Iron, but that's exactly what it was doing.

9

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.