Rust’s current approach to evaluating optimizations are:
- I did it and now rustc builds faster/smaller
#[bench]I made just for this seems to do better
The first is problematic because rustc is neither idiomatic nor all-encompasing. The latter is problematic because micro-benching correctly is super hard, and even if you do it right, there might be unintended consequences for a different workload.
Therefore I believe we need to collect up a selection of idiomatic non-trivial pieces of code to more robustly evaluate optimizations against. Things that upon inspection should perform well, or at least are “how you would do it”. Heck, maybe even some examples of common mistakes or bad idioms. Things that leverage lots of functionality. Things that we can track and perhaps gate against.
The benchmarks game files are perhaps the closest things we have to this today, but they’re a bit tainted by the fact that they’re written by us to make Rust look good, rather than get a job done.