Custom CLI for benchmarks on top of the one from Criterion

help
1

Hi everyone,
I am having some trouble implementing a custom clap CLI which handles specific options for my benchmarks, while also having the commands available for Criterion.
The problem is that Criterion doesn't allow to parse only specific arguments from the command line, it captures all of them using the configure_from_args() function.

I thought of capturing the criterion arguments first and then putting as trailing options my custom arguments after --. This way, I could access for example the help of Criterion using:

cargo bench -- --help

and instead, access my custom help using:

cargo bench -- -- --help

if that makes sense. But I don't really know how to do this with clap. Can someone help me?

Finally, by fiddling around I encountered an unexpected behavior of cargo bench. It always adds the flag --bench at the end of the command line arguments, even if I don't provide anything to the command line. Does anyone know why?

2

This won't work, because you'll get an error from Criterion's argument parser about the unexpected arguments; it won't just ignore them. Testing with one of my own:

$ cargo bench --bench raytrace -- -- --foo --bar
    Finished `bench` profile [optimized] target(s) in 0.51s
     Running benches/raytrace.rs (target/release/deps/raytrace-12ec0e5446510037)
error: unexpected argument '--bar' found

Usage: raytrace-12ec0e5446510037 [OPTIONS] [FILTER]

I think your best option here is:

  • Send a PR to Criterion so it can parse args from an arbitrary iterator.
  • Until you can make use of that, configure your benchmark using environment variables instead of args.

In the convention set by cargo and rustc, and followed by criterion, benchmark binaries and test binaries are largely the same thing — they both link against libtest and use its entry point. If you don't pass --bench, then the binary runs tests, and runs benchmarks once to confirm they succeed, instead.

(In order to actually use this behavior from Cargo, you'd run cargo test --all-targets or cargo test --benches.)

2 Likes
3

Thank you! Yeah, I was worried there was nothing much to do about it. I'll try your suggestions!

4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.