Useful tip for benchmarking/testing optional Cargo features

The following is helpful for anyone, who is struggling with:

  • creating a crate
  • that has optional features
  • and criterion benchmarks that require these optional features

Today I learned that in Rust...
...when you are using Cargo features together with criterion in your crate and your benchmark requires a certain feature that is not a default feature, you have to tell the benchmark about it in your Cargo.toml.

Let's take the following folder structure of a crate with criterion benchmarks as an example:

.
└── my_crate
    ├── src
    │   └── lib.rs
    ├── benches
    │   └── my_benchmark.rs
    └── Cargo.toml

Let's now say we optionally depend on rayon, which makes it a feature, others can opt-in to.
Extract from Cargo.toml:

...

[dependencies]
rayon = { version = "1.5", optional = true }

[dev-dependencies]
criterion = "0.3"

We also need to let cargo know of our benchmark, so let's follow the criterion docs:

[[bench]]
name = "my_benchmark"
harness = false

Our lib.rs looks something like this:

pub struct MyStruct;

impl MyStruct {
    pub fn do_something_single_threaded() {
        println!("something single-threaded");
    }
    
    #[cfg(feature = "rayon")]
    // we want to benchmark this function, but this is only available,
    // if the consumer of the crate opt-in into the rayon feature
    pub fn do_something_multi_threaded_with_rayon() {
        println!("something multi-threaded with rayon");
    }
}

And now we come to our my_benchmark.rs file, which could look something like this:

use criterion::{criterion_group, criterion_main, Criterion};
use my_crate::MyStruct;

pub fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("benchmarking our multi-threaded function that depends on rayon", |b| {
        b.iter(|| MyStruct::do_something_multi_threaded_with_rayon())
        //                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        //                  this will lead to a compile error, because we need
        //                  the "rayon" feature for it
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

But wait!? This won't compile, because our associated function is behind the "rayon" feature flag. :open_mouth:
Meh, that's a shame. :disappointed: How can we make the Rust compiler happy again? We have become such good friends after all, right!? :pleading_face:
We somehow need to tell it that the feature "rayon" is required for the benchmarks.

Well, in order to do that, we just have to add the property required-features to our Cargo.toml in the [[bench]] section:

[[bench]]
name = "my_benchmark"
harness = false
required-features = ["rayon"] # this is the new property

Yay, Rust compiler is happy again (and we are, too :grin:)! :hugs:

And now, to start your benchmark, execute:
cargo bench --features rayon

which tells cargo to use the feature "rayon" for your benchmark (if you omit --features rayon) there won't be an error, but your benchmark just won't execute.

On a side-note: You can use required-features also for your integration tests ([[test]] section in Cargo.toml).


I hope this helps anyone, who comes across this.
If you spot any errors, please let me know.

Big thank you to the authors of tinyvec, which has acted as a reference for me on how to do this.:heart:

Sorry for reviving such an old thread, but I think a "Today I learned in Rust" is very useful and I didn't want to create a duplicate.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.