Hi rustaceans. I have some code which i'm benchmarking using cargo +nightly bench. The function i'm benchmarking (compiling a shader) takes 5-10 seconds. Cargo's benchmark runner doesn't seem to react intelligently to this, as it keeps running this benchmark for 5-10 minutes before it finally exits and prints the benchmark results.
I see in the test crate's code there is already support for running benchmarks once instead of multiple times, but i see no documented way to trigger this behavior.
How can I use cargo bench to benchmark this code without having to wait 5 minutes every time I want to see a result?
The (unstable) benchmark support in cargo bench is very minimal. Most people use a third-party benchmark framework like criterion or divan. Both of them allow configuring the number of iterations, and criterion, at least (I haven't personally tried divan), will adapt to the speed of the function, within limits.
However, if you reduce it to 1 run then your results will be noisy, and you won't get very much benefit from using any benchmarking framework, so, if that's really what you have to do, then consider writing your own code entirely — set bench.your_bench_name_here.harness = false in the Cargo.toml and write a main() that runs your functions once or however you like. You have total control.
With Criterion, it is easy to set the run time, sample size and warm up time. In my case, it reported that my original 15 s measurement time is too small for some tests for all samples to complete, so I simply raised to larger. Criterion seems measuring the number of samples per time, not the duration of each sample.