SIGILL: illegal instruction when running cargo test

#1

Hi,

when I run cargo test on this repository
https://github.com/hyperledger/indy-sdk/ the process stops with SIGILL.

To reproduce:

  1. git clone https://github.com/hyperledger/indy-sdk/
  2. cd libindy
  3. cargo test

Process didn’t exit successfully (signal: 4, SIGILL: illegal instruction)

To be clear, I expect the tests to fail because they are using global resources which does not work because cargo tests are running in parallel. But I didn’t cargo to crash.

I am on Ubuntu 16.04. rustc version is 1.33.0

Axel

#2

Do you have any more information, like maybe a backtrace? I can’t compile this on my machine because I don’t have libsodium.

I see a bunch of unsafe code in this package, I would bet that something is wrong somewhere in there. If it’s incorrect, and depending exactly on what you mean by “using global resources which does not work”, then you’re probably triggering UB, which I would expect to crash.

#3

You can affect this by setting RUST_TEST_THREADS=1, which indy-sdk mentions in the build guides. You can also use cargo test -j1, or use --test-threads 1 directly on the test binary.

#4

I managed to reproduce the failure in GDB – here’s the backtrace:

#0  0x000055c9d8ad06af in std::panicking::rust_panic_with_hook ()
#1  0x000055c9d8ad0112 in std::panicking::continue_panic_fmt ()
#2  0x000055c9d8acfff6 in rust_begin_unwind ()
#3  0x000055c9d8ae03fd in core::panicking::panic_fmt ()
#4  0x000055c9d84de913 in core::result::unwrap_failed (msg=..., error=...)
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/macros.rs:16
#5  0x000055c9d84b2890 in <core::result::Result<T, E>>::unwrap (self=...)
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/result.rs:798
#6  0x000055c9d81500d0 in <indy::services::wallet::storage::plugged::PluggedStorage as core::ops::drop::Drop>::drop (
    self=0x7f01d40098b0) at src/services/wallet/storage/plugged/mod.rs:632
#7  0x000055c9d7c6094e in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#8  0x000055c9d7c57d87 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#9  0x000055c9d7c54e6a in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#10 0x000055c9d7c607b4 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#11 0x000055c9d7c4bf62 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#12 0x000055c9d82ad7b4 in core::ptr::drop_in_place (to_drop=0x7f01d402ae70)
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:183
#13 <std::collections::hash::table::RawTable<K, V>>::rev_drop_buckets (self=0x7f021728bfe8)
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libstd/collections/hash/table.rs:837
#14 0x000055c9d838aaef in <std::collections::hash::table::RawTable<K, V> as core::ops::drop::Drop>::drop (
    self=0x7f021728bfe8) at /builddir/build/BUILD/rustc-1.33.0-src/src/libstd/collections/hash/table.rs:1120
#15 0x000055c9d7c5f49e in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#16 0x000055c9d7c5f612 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#17 0x000055c9d7c6d83e in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#18 0x000055c9d7c74cb2 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#19 0x000055c9d7c65842 in core::ptr::real_drop_in_place ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ptr.rs:193
#20 0x000055c9d7b9edcc in indy::services::wallet::tests::wallet_service_add_record_works_for_plugged ()
    at src/services/wallet/mod.rs:1139
#21 0x000055c9d7dd9aca in indy::services::wallet::tests::wallet_service_add_record_works_for_plugged::{{closure}} ()
    at src/services/wallet/mod.rs:1128
#22 0x000055c9d7c3c70e in core::ops::function::FnOnce::call_once ()
    at /builddir/build/BUILD/rustc-1.33.0-src/src/libcore/ops/function.rs:231
#23 0x000055c9d86191af in <F as alloc::boxed::FnBox<A>>::call_box ()
#24 0x000055c9d8ad318a in __rust_maybe_catch_panic ()
#25 0x000055c9d863931e in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#26 0x000055c9d8637585 in std::panicking::try::do_call ()
#27 0x000055c9d8ad318a in __rust_maybe_catch_panic ()
#28 0x000055c9d863aa1d in <F as alloc::boxed::FnBox<A>>::call_box ()
#29 0x000055c9d8ace5fe in std::sys_common::thread::start_thread ()
#30 0x000055c9d8ac0c26 in std::sys::unix::thread::Thread::new::thread_start ()
#31 0x00007f0217a5758e in start_thread () from /lib64/libpthread.so.0
#32 0x00007f021796b6a3 in clone () from /lib64/libc.so.6

In particular, at frame #6:

#6  0x000055c9d81500d0 in <indy::services::wallet::storage::plugged::PluggedStorage as core::ops::drop::Drop>::drop (
    self=0x7f01d40098b0) at src/services/wallet/storage/plugged/mod.rs:632
632                 self.close().unwrap();

That’s an unwrap() in Drop – if this fails while drop is called for panic-unwinding, then you’ll have a double-panic, which aborts. Low-level aborts like this are often implemented with a “bad” instruction, here a ud2: “Generates an invalid opcode. This instruction is provided for software testing to explicitly generate an invalid opcode. The opcode for this instruction is reserved for this purpose.”

Normally, Rust does print a last-ditch error message too, but tests capture their output by default to try and present it nicely for test failures. You can pass --nocapture to override this, and then you’ll see a backtrace and “thread panicked while panicking. aborting.” before the ud2 abort.

Hmm, I wonder if we can make the double-panic error message bypass the captured output, and instead write stderr directly.

5 Likes
#5

Thanks! BTW: I know how to run the tests in one thread. I am trying to get rid of this limitation so that the tests can run in parallel.

The “plugged” stuff is implemented in inmem_wallet.rs and I assume that in the end that “close” is called. https://github.com/hyperledger/indy-sdk/blob/master/libindy/src/utils/inmem_wallet.rs#L732

Maybe the lock can’t be acquired at
let mut handles = INMEM_OPEN_WALLETS.lock().unwrap();

Any suggestions on how to improve the code?

Thanks again
Axel

#6

regarding the unsafe code: That is because the main thing this crate does is export a c-callable API.

So when the tests call this API they turn the arguments to ctypes and wrap the call into unsafe.
I assume there is no way around this.

Would be nice if Rust code could call the API and still have Rust’s guaranties.
This is a rare crate that is written in Rust but exposes a C callable API.
The repo does have Rust wrappers and a indy-sys but that is generated manually.

#7

It may make more sense to have a clean Rust API, leaving the unsafe entry points just for FFI callers.

That’s what librsvg is doing: https://people.gnome.org/~federico/blog/a-rust-api-for-librsvg.html

#8

It is probably too late for that. I think when the project started bindgen was not in a stable state and creating the C-Bindings and comparing them with the expected binding was hard. So there we are…

Thanks for the librsvg example. Interesting. (That is a German genuine interesting)