How to benchmark multiple versions of a library (with different compile-time constants) against each other?

I am working on a small research project at the university, about parallelizing L-system generation.

The 'fun' part of this will of course be to implement a couple of algorithms to do this in Rust.
And then these should be compared to each-other by benchmarking.

So far I've seen the wonderful library Criterion which seems to have many of the bells and whistles that I require.

There is only one thing that I have not yet figured out:
I want to also test an algorithm implementation against 'itself' with slightly altered constants. For instance, for 'small' collections I want to perform work on a single thread, whereas when a collection is larger than a threshold size, work should be split across multiple threads.
But how big should this threshold be? 128 elements? 1024 elements? smaller? bigger? That's one of the things I want to figure out by benchmarking.

However, how to set this up? I know Criterion treats the code used in the benchmarks as separate crates, but is there a way to compile my crate multiple times with different compile-time settings (like above 'threshold')?

Or is there another approach that I should use instead of Criterion for this?

2 Likes

Out-of-Curiosity, why would that type of "Constant" need to be a compile-time constant instead of a command-line/configuration parameter?

Because I think its value might directly affect how the code will be compiled (i.e. how or if code can be optimized at the LLVM level). What most immediate comes to mind is whether LLVM can use certain kinds of SIMD-style optimizations for certain loops.

This seems like something you could do with git and maybe a little shell scripting.

Make a topic branch with the alternative algorithm. Any time you make changes to the main branch, port them over to the topic branch as well. You could automate the test runner to check out the main branch, run the benchmarks, then check out the topic branch, and run the benchmarks again.

I've done this using Criterion, by pretending that Rust has integer generics.

That is, you'd really like to be able to compare MyType<100> against MyType<1000>, but this would require a language feature called "const generics" that isn't stable yet.

You can fake it using typenum, a crate which provides type-level representatives of integers (like typenum::N4 instead of 4) and the ability to obtain the equivalent integer using a const fn.

You then need to make all your tweakable parameters into actual generic parameters on the types, so that the values can be provided.

1 Like

Thank you for your feedback!

I currently think that typenum will fit well for my use-case, so that is the direction I will try out :slight_smile:.

https://doc.rust-lang.org/stable/std/macro.option_env.html

1 Like

Interesting! How would something like this be used in combination with e.g. Criterion? That would mean that the same crate has to be compiled/included multiple times with different options being passed. :thinking: hmm...

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.