I currently use criterion to run benchmarks. Most of the time, what I care about is the change in run time between some prior revision of the code and whatever I just wrote. criterion does support comparison against the previous run and against “baselines”, but:
I have to explicitly save a baseline for the current state of my upstream branch, and it won’t notice if I forget to update the baseline due to pushing changes. Ideally, every remembered benchmark result would be associated with a revision (and build flags, too, while we’re at it).
Since I have no dedicated machine for benchmarking, my results are always somewhat unstable, and it’s difficult to tell whether a difference is due to the code or due to the conditions under which I run it. Ideally, the benchmark tool would automatically run both versions interleaved (by building both versions and running both binaries), thus keeping the comparison fairer.
cargo-criterion is the new way to run criterion, but doesn’t support baselines. Also, criterion development has been slow to nonexistent for a long time.
Is anyone aware of tools or strategies for benchmarking that can better support this kind of workflow? I’m feeling tempted to write my own.
This in particular is a fantastic idea. Suppose you are benchmarking for run-time. Benchmarking results will be reported in (e.g.) seconds. The problem is that this is a "dimension-full" quantity, and so is tied to the environment in which it was run. What is needed is an absolute number without dimensions ... a dimensionless quantity. But performing two runs as you describe and (for example) dividing one by the other, you've now got an absolute number without dimensions. Absolute numbers are things you can (almost) always compare with reckless abandon.
A couple of additional thoughts.
Of course, one doesn't have to go through the whole rigmarole of doing the division. Just comparing the values it likely enough.
I don't believe it'll ever be possible to completely eliminate the "environment" factor. If the run uses versions A and B, and shows that A was faster than B, maybe the benchmarking environment just happens to do a better job of executing A. Does that necessarily mean that A is preferable to B? But by comparing two runs like this, the environment factor is greatly reduced IMHO.
Maybe I'm reading too much into what you said, but I also think the idea of storing benchmark results against commit hashes is a good one. Sounds like a space efficient mapping.
