How to run FFI tests individually

[ehsanmok]

Hi

I'm writing a Rust binding and there could be some thread sanitization issues inevitably accessing some global items from .so or registering some global functions that when I run my tests even with --test-threads=1 at least one would fail but if I run each individually by name cargo test myfunction then they all pass.

Is there any way to run tests individually in this case besides adding unwanted test_features = [] or something?

[azriel91]

You could try using a Mutex around each test, @durka posted this on github:

use std::sync::Mutex;
#[macro_use] extern crate lazy_static;

lazy_static! {
    static ref TEST_MUTEX: Mutex<()> = Mutex::new(());
}

macro_rules! test {
    (fn $name:ident() $body:block) => {
        #[test]
        fn $name() {
            let guard = $crate::TEST_MUTEX.lock().unwrap();
            if let Err(e) = panic::catch_unwind(|| { $body }) {
                drop(guard);
                panic::resume_unwind(e);
            }
        }
    }
}

test! { fn one() { println!("one"); } }
test! { fn two() { println!("two"); } }

[ehsanmok]

Thanks! but unfortunately it doesn't work my case.

[jethrogb]

@ehsanmok

Thanks! but unfortunately it doesn’t work my case.

Why not?

[Michael-F-Bryan]

@jethrogb

Why not?

It sounds like the library being wrapped plays around with some global state, so after the first test fails the world is left in an inconsistent state and all later tests blow up as well.

It sounds like instead of restricting tests to one thread, you want to run each test in its own process. It shouldn't be too hard to knock up a python/bash script (or maybe even a Rust program under the tests/ directory) which looks for the my_test bit in #[test]\nfn my_test() and invokes each test with cargo test my_test... It'll take forever to run your test suite that way though...

[jethrogb]

@Michael-F-Bryan

It sounds like the library being wrapped plays around with some global state, so after the first test fails the world is left in an inconsistent state and all later tests blow up as well.

It sounds like instead of restricting tests to one thread, you want to run each test in its own process.

This makes sense now. I'm now thinking about some way to do this in the test runner with fork and IPC but your way is probably more sane.

[Michael-F-Bryan]

@jethrogb

This makes sense now. I’m now thinking about some way to do this in the test runner with fork and IPC but your way is probably more sane.

What about writing your tests without rustc inserting its default test harness? That way you can write a normal Rust program intests/foo.rs to invoke the FFI tests and it'll be executed whenever you run cargo test.

@SimonSapin

harness = false

The test crate is unstable, and likely to be for a while. It’d be good to experiment a bit on crates.io . To support this, Cargo allows specifying in Cargo.toml :

[[test]]
name = "foo"
harness = false

(Note: when you do this Cargo stops enumerating tests/*.rs implicitly and you need an explicit [[test]] section for every test file.)

This causes cargo test to not rustc --test when compiling that test file and instead compile it as an executable with a main() function. Then it’s up to you to test stuff and exit a non-zero status code on failure.

(source)

[ehsanmok]

Yes, that's right! in particular, there's a module which works with globally registered functions in .so and it's also possible to globally register functions. Now my unit tests for that module pass or fail underministically and using mutex for each unit test didn't help either.