Using custom_test_frameworks without using cargo test

help
1

Hello all,

I am writing rust code that has only one purpose - to run sets of tests on a (pre-existing) binary. For various reasons, I'm using the custom_test_frameworks feature, and invoking the tests via $ cargo test. This runs the tests fine, but there are two problems with it:

  1. I still see cargo test's standard output, which I don't really want, like:
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  1. (more signficantly) I can't find a way to pass command-line arguments to the test runner. If I invoke the tests via cargo test -- --custom-arg-for-test-runner, then the test runner does receive this argument (via std::env::args()), but cargo test tries to parse it as an argument for itself, and so errors with an unrecognized argument error.

I've explored a few avenues for dealing with this:
a. Invoking the test_runner from main() in src/main.rs, so I can do a test run via cargo run rather than cargo test
b. Building the tests executable via rustc rather than cargo, then running it directly
c. Using some other method of collecting all the test functions, rather than the #[test_case] attribute that custom_test_frameworks provides
... but none of these have entirely worked out and it's not obvious to me which is the most promising.

Any guidance appreciated.

2

I haven't tried custom_test_frameworks, but cargo test by default runs all the tests in the current package, which means it'll make a test binary for each of the library, any binaries, any examples, any integration tests, and the library's doctests. Arguments are passed to all the test binaries.

It sounds like the custom framework replaces the library test binary, but not the doctests or other targets, so you would need to pass cargo test --lib -- --custom-arg-for-test-runner to only run your custom framework.

3

I can't see that this works - I get this from cargo test if I try it:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running unittests src/lib.rs (target/debug/deps/e2e_tests-87a59006d23797b3)
error: Unrecognized option: 'custom-arg-for-test-runner'
error: test failed, to rerun pass `--lib`
4

It's totally possible that custom_test_frameworks isn't able to handle extra options yet. But other than that, I'd probably need more info about the framework and project to tell what's going on.

5

I think you still have a test which uses libtest rather than custom_test_frameworks. Likely this is the library target (src/lib.rs). You can either test just the binary using cargo test --bin e2e_tests -- --custom-arg-for-test-runner assuming your binary is called e2e_tests or you can disable testing for the library target by adding the following to Cargo.toml:

[lib]
test = false
1 Like
6

Thanks! It was exactly this. It works well now.

7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.