Is there a way to get `cargo test` to emit a backtrace on interrupted tests?

1

I'm currently debugging a deadlock in some concurrent code. Due to some interactions between static variables and execution order, running cargo test deadlocks once in every ten runs or so. Typically, I just then interrupt cargo test with Ctrl+C.

I'd like to be able to get a stack trace of when I interrupt the code so that I can debug my deadlock, but simply running RUST_BACKTRACE=1 cargo test doesn't emit a backtrace when canceled. Is there a setting for cargo test to make this possible, and if not, is there a better way to debug and detect deadlocks?

2

Adding -- --nocapture may help.

And, trying to isolate the troublesome test, by running one test at a time, should help.

3

Sadly, this is a deep concurrency bug that seems to require multiple tests to be running to reproduce. Since tests run as multiple threads in the same process, and my code involves manipulating static locks, it's actually the concurrent interaction between tests that causes the issue.

4

Attaching a sampling profiler to the process, after it is deadlocked, will get you the call stacks you want.

(If you happen to be on macOS, there is one built in, no additional tools needed — you can access it from Activity Monitor or use the sample command-line tool.)

2 Likes
5

Thanks! I'm on linux, so I ran the test binary with gdb and then did thread apply all bt to get backtraces for each running thread.