How to properly add test cases for benches?

help
1

I have some benches, I want to ensure that all the benched functions behave correctly, so I will write some test cases with them.

I use criterion and the main bench file is

use {
    crate::{
        a::bench_a,
        b::bench_b,
    },
    criterion::{
        criterion_group,
        criterion_main,
        Criterion,
    },
};

mod a;
mod b;

criterion_group!(benches, bench_all);
criterion_main!(benches);

pub fn bench_all(c: &mut Criterion) {
    bench_a(c);
    bench_b(c);
}

I find that when I add test cases in a.rs or b.rs, if I import something, the linter/compiler complains "unused import", but if I delete it, the linter/compiler complains "unresolved reference". At last, I have to make all import, modues(include a and b) and functions public to avoid warnings.

#[cfg(test)]
pub mod test {
    pub use std::error::Error;
    pub use x::y::z;
    #[test]
    pub test_f() -> Result<(), Box<dyn Error>> {
        z()
    }
}

So is this a bug of linter/compiler? How can I properly add test cases for benches?

2

Can you share the version of the code that does not have pubs and produces "unused import" warnings? Generally all that you should need to do is make sure your test-only imports are inside the test module.

3

The version of the code that produces unused import is the same to the code above except removing any of the pub (mod a, mod test, use std::error::Error, use x::y::z or fn test_f)

4

I notice that you're using out-of-line modules. Is your file tree layout like this?

Cargo.toml
benches/
    my_bench.rs
    a.rs
    b.rs

If so, that would be expected to produce spurious warnings, because a.rs and b.rs are being detected as bench crates by Cargo. You should use this file layout instead (standard for multi-file targets):

Cargo.toml
benches/
    my_bench/
        main.rs
        a.rs
        b.rs

If that's not it, can you please show your actual file list, and also the full text of the warnings produced by cargo check? That will give more clues as to what the actual problem is.

3 Likes
5

Yes, my file tree layout like that. I didn't know that there is some way to define multi-file target before, so thank you! But unfortunately changing the file layout as you suggest doesn't fix the problem -- there are still unused imports warnings. And I find out that when running cargo test --all-targets, the test cases defined in a.rs and b.rs are not run, which was run under the old file layout. So maybe only test cases defined at top bench modules will be run?

6

btw, my rustc version: rustc 1.76.0-nightly (a57770440 2023-11-16)

7

Can you please post the full output from cargo, with all the details, as well as the exact command you ran to cause it? That will give useful information about what is going on.

8

the project: https://github.com/LambdaAlpha/airlang_rs/tree/dev/lib/benches
the command: cargo test --all-targets

  1. old file layout with all pubs
Compiling airlang v0.0.8 (/home/lqh/arepo/Projects/airlang_rs/lib)
    Finished test [unoptimized + debuginfo] target(s) in 1.99s
     Running unittests src/lib.rs (target/debug/deps/airlang-086801d20cc8a704)

running 66 tests
test semantics::test::test_bytes ... ok
`and so on`

test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

     Running benches/bench_main.rs (target/debug/deps/bench_main-ced8b5e09ccd4501)
Gnuplot not found, using plotters backend
Testing parse
Success

Testing generate
Success

Testing interpret
Success

Testing semantic parse
Success

Testing semantic generate
Success

     Running benches/semantics.rs (target/debug/deps/semantics-b774793d13be8b81)

running 3 tests
test test::test_parse ... ok
test test::test_generate ... ok
test test::test_interpret ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/syntax.rs (target/debug/deps/syntax-5bac985205f29286)

running 2 tests
test test::test_parse ... ok
test test::test_generate ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/airlang_bin-d4748f6ad55128b1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  1. old file layout without any a pub
Compiling airlang v0.0.8 (/home/lqh/arepo/Projects/airlang_rs/lib)
warning: unused imports: `Interpreter`, `Val`, `generate`, `parse`, `std::error::Error`, `types::Int`
  --> lib/benches/semantics.rs:53:17
   |
53 |                 generate,
   |                 ^^^^^^^^
54 |                 parse,
   |                 ^^^^^
55 |                 Interpreter,
   |                 ^^^^^^^^^^^
56 |                 Val,
   |                 ^^^
57 |             },
58 |             types::Int,
   |             ^^^^^^^^^^
59 |         },
60 |         std::error::Error,
   |         ^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: `airlang` (bench "bench_main") generated 1 warning (run `cargo fix --bench "bench_main"` to apply 1 suggestion)
    Finished test [unoptimized + debuginfo] target(s) in 1.94s
     Running unittests src/lib.rs (target/debug/deps/airlang-086801d20cc8a704)

running 66 tests
test semantics::test::test_bytes ... ok
`and so on`

test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

     Running benches/bench_main.rs (target/debug/deps/bench_main-ced8b5e09ccd4501)
Gnuplot not found, using plotters backend
Testing parse
Success

Testing generate
Success

Testing interpret
Success

Testing semantic parse
Success

Testing semantic generate
Success

     Running benches/semantics.rs (target/debug/deps/semantics-b774793d13be8b81)

running 3 tests
test test::test_parse ... ok
test test::test_generate ... ok
test test::test_interpret ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/syntax.rs (target/debug/deps/syntax-5bac985205f29286)

running 2 tests
test test::test_parse ... ok
test test::test_generate ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/airlang_bin-d4748f6ad55128b1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  1. new file layout with all pubs
   Compiling airlang v0.0.8 (/home/lqh/arepo/Projects/airlang_rs/lib)
    Finished test [unoptimized + debuginfo] target(s) in 1.85s
     Running unittests src/lib.rs (target/debug/deps/airlang-086801d20cc8a704)

running 66 tests
test semantics::test::test_bytes ... ok
`and so on`

test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

     Running benches/main/main.rs (target/debug/deps/main-cb32bd1904cb562d)
Gnuplot not found, using plotters backend
Testing parse
Success

Testing generate
Success

Testing interpret
Success

Testing semantic parse
Success

Testing semantic generate
Success

     Running unittests src/main.rs (target/debug/deps/airlang_bin-d4748f6ad55128b1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
  1. new file layout without any a pub
Compiling airlang v0.0.8 (/home/lqh/arepo/Projects/airlang_rs/lib)
warning: unused imports: `Interpreter`, `Val`, `generate`, `parse`, `std::error::Error`, `types::Int`
  --> lib/benches/main/semantics.rs:53:17
   |
53 |                 generate,
   |                 ^^^^^^^^
54 |                 parse,
   |                 ^^^^^
55 |                 Interpreter,
   |                 ^^^^^^^^^^^
56 |                 Val,
   |                 ^^^
57 |             },
58 |             types::Int,
   |             ^^^^^^^^^^
59 |         },
60 |         std::error::Error,
   |         ^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: `airlang` (bench "main") generated 1 warning (run `cargo fix --bench "main"` to apply 1 suggestion)
    Finished test [unoptimized + debuginfo] target(s) in 1.85s
     Running unittests src/lib.rs (target/debug/deps/airlang-086801d20cc8a704)

running 66 tests
test semantics::test::test_bytes ... ok
`and so on`

test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

     Running benches/main/main.rs (target/debug/deps/main-cb32bd1904cb562d)
Gnuplot not found, using plotters backend
Testing parse
Success

Testing generate
Success

Testing interpret
Success

Testing semantic parse
Success

Testing semantic generate
Success

     Running unittests src/main.rs (target/debug/deps/airlang_bin-d4748f6ad55128b1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9

OK, we're making progress. Can you please publish a branch that contains this version of the code, the one where you rearranged the bench files according to my recommendation and got the following error?

10

Actually, never mind that. I understand the problem.

The problem is that benchmarks are built with cfg(test) enabled, so your mod test is compiled when compiling the benchmark.

The best solution I can think of here is to eliminate the mod test module entirely (it's not giving you any benefits) so that the tests can make use of the same use as the rest of the code does.

You could also make your project layout less unusual by moving your test code to tests/ instead of combining it with your benchmarks. It doesn't seem to be actually testing the benchmarking code, just the library (but perhaps I misunderstand).

2 Likes
11

I'm not test the benching code, I'm testing the function being benched. It should behave correctly on given input. I actually have many test cases for the function, but I want to ensure that when benching, the function won't fail fast and we won't get a false benching result. I want to ensure that my changes to code won't break the benching cases.

12

I just move the test cases for benches to tests directory to fix this problem.

pros:

  • no warnings now.
  • cargo test --tests will run the tests, the bench target won't be built, which may save building time.

cons:

  • The codes related to each other are not placed together.
  • I need to share some resource between benches and tests.