How does `rustc` get accurate benchmarking data

[cameron1024]

I'm looking to run some benchmarks against a Rust program as part of a CI pipeline, but I've noticed pretty massive fluctuations in the performance of the underlying cloud infrastructure I'm running on.

In comparison, looking at rustc performance data , seems largely consistent.

Is there any documentation/reading I can do to learn about how to set up a stable environment in the cloud for benchmarking Rust binaries? I appreciate this is more of an infrastructure question than a Rust question per se, but hoping someone here can point me in the right direction.

Most of my googling either brings up fairly opaque academic papers, or comparisons between major cloud infrastructure providers

Any help would be much appreciated

[bjorn3]

@cameron1024

Is there any documentation/reading I can do to learn about how to set up a stable environment in the cloud for benchmarking Rust binaries?

Use a bare machine rather than the cloud. Noisy neighboars are unavoidable in the cloud by design. Even worse, performance between two identical machines may differ a bit. The only way to get consistent benchmark results is to run on all benchmarks you want to compare the same bare metal server. This is what rust does for perf.rust-lang.org.

[cameron1024]

Thanks for the info, I was slightly worried this would be the answer. I guess time to do things the old-fashioned way

[Michael-F-Bryan]

It doesn't necessarily need to be done manually.

Most CI systems let you register custom runners, so you could (for example) create a job in GitHub Actions which will only ever run on a dedicated machine using labels, then make sure nothing else runs at the same time by limiting concurrency.