For a long time I have wondered why when you try to measure a relatively small time something takes to execute, the results vary wildly. I think I have just figured out what is probably happening, and thought I would share it.
"So this CPU has frequency ranges of 3.4 to 3.7GHz, 2.9 to 3.6GHz or 2.3 to 2.9GHz when operating on all cores....."
"An idle CPU will reduce its clock-speed, and operating voltage, quite dramatically. The clock speed is generally reduced to around 1GHz (or 2GHz for AMD), and then restored relatively rapidly when the CPU stops being idle. I would recommend leaving this power-saving on: it saves electricity when the machine is idle, and the small fraction of a second (under 1ms) taken to restore the CPU's speed at the start of a benchmark is not very significant compared to the usual program start-up overheads."
So if you try to measure something that lasts about 1msec or less, depending on when the clock speed is increased, you will see a large variation in the time to do the same thing. Well I think that is the answer, I cannot prove it.
it's even worse, the clock speed of the CPU depends on its temperature (if its too hot it throttles to not overheat), so if your cooling is not great and the CPU is on the edge of overheating during the winter your benchmarks will give worse results when its summer.
This is one of the reasons why just taking one data point is usually an entirely useless benchmark. You need a larger sample and some actual statistical methods.
Maybe, but perhaps the way to do benchmarking is to warm up until processor is going maximum speed (may take from 1.6ms up to 564ms), check the speed (and temperature), do the test(s) [ just once - no need for lots of stats], check the processor speed is still the same, then stop. Rather than spending minutes doing the same thing over and over again for no real benefit?
another possibility is if your computer's cooling system is less-than-great but has a lot of heat capacity then it may slowly saturate with heat and eventually result in thermal throttling in later test runs.
My chromebook has a screen where you can see processor temperature and current speed (also the battery current), which is quite useful. But definitely best to check the temperature before attempting any benchmarking. I don't know if any benchmark crates do that. If you have say 100 benchmarks to run, I find some of the current benchmark crates a bit slow to use.
They don't do that because that's not how you get stable results. If you want to have somewhat reproducible results then you need to switch ChromeBook into development mode, disable P-states, isolate CPU for that benchmark (Linux kernel command line intel_pstate=no_hwp isolcpus=2,3 would do that), and so on.
You still need to disable turbo boost with echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo and disable kernel-driven frequency ramp-up with adb shell “echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor”.
But that's simply “ChromeOS benchmarking 101”, it's not related to Rust and, more importantly, can not be automatically done by rust benchmarking crates, thus, of course, it's your responsibility to prepare test device for benchmarking.
Are you sure getting stable results depends on running tests many times? Where does variability come from other than the clock speed changing? I have found the existing benchmark tools to be rather poor when it comes to getting stable results, although I haven't done much recently (I did try using them before I had a chromebook, that was a windows laptop). The problem I remember is you change something which maybe gives a 2% improvement, but the benchmark tool results have too much variability to detect that reliably. It was quite frustrating.
There are bazillion sources. Things like SMM can not be disabled, on modern hardware. Heck, even memory access these days requires training and may have different speed from one boot to another.
Windows is entirely hopeless. ChromeOS can be brought to kinda-sorta quiescent state where 1% difference can be noticed on a test devioce (not device where you are doing development, of course), Windows is almost impossible to bring to that state, after boot you already have hundreds of background processes that are doing various things.
2% is entirely impossible to notice if you haven't prepared your system for that. 10% is possible to notice even on Windows, but only if your benchmark does enough repetitions and you disabled all kinds of boost in BIOS Setup.
I just tried running some old criterion tests, the CPU temperature goes up to 94C ( higher than I have normally seen it, 40C-60C is normal) and the clock speed varies considerably as it runs, presumably because running flat out for seconds at a time isn't sustainable. It is like having an Olympic games 100m final, but insisting the competitors run a marathon before and after!
So the results are rather useless, you get different results for each criterion run. Maybe I could twiddle some criterion knobs to just run each test once or twice and that would give much better results, I am just using the default settings. I will have a go tomorrow.
It's perfectly sustainable if you disable P-states and Turbo boost. Of course it would mean that your device would be significantly slow thus it may only be done if you boot device in dev mode and specify appropriate kernel argument.
Not if you prepare your device properly. Doing that on any portable device is non-trivial, though. Most of the time if you are serious about performance work you want to use dedicated workstation that doesn't try to pair chip that produces 100-200W or thermal load with 10-20W cooling system.
How much value is there in measuring times like this? Won't it just give you a specific point of data for your specific hardware setup, software setup and compiled binary? Most code doesn't run in such controlled environments does it?
To my understanding benchmarks are mainly useful to double check your assumptions and validate the performance impact of changes you make. If your old code is slow but your new code is faster but only because it makes the CPU power up isn't that useful data showing you what might happen in the real world?
I haven't found a way to do that on a Chromebook yet. I am not aware of any kind of BIOS setup or similar or how to get to it. I don't want to break it! There is "developer mode", but entering it wipes everything off your system, so not very practical. I did try battery saving mode, but the clock speed is still unstable.
Another approach might be to have two threads, one to "count clocks" (as described in the link in my post), the other to do the test. The result is then how many clocks the test took, which might be much more stable ( rather than results varying by a random 10% each time the test is done ). I cannot think quite how to program that though, how would the clock-counting thread know exactly when the test thread is finished? I guess it could continuously poll an atomic or something, maybe that would be good enough. [ Edit: or maybe just try several times, adjusting the loop iterations for the clock-counting thread until the termination times match. Then record that as the bench result, and compare it with the result after making some code change. ]
Typically you are making some change you hope will speed things up, and want to know what effect the change has had, but the unstable clock speed (which as far as I can tell varies wildly and unpredictably) makes it hard to get reliable results in a reasonable amount of time for small changes ( say a 2% improvement ). If you make 10 x 2% improvements, the overall effect should be good.
If you make lots of small improvements, the cumulative effect should be significant. Also if you run for long enough, the noise effects will average out. The problem is a practical one, if it takes 3 days to figure out if a small change is good, that is not practical. In practice I remember doing the benchmark quite a few times, try to get an impression if a change is helping, but you end up in a state of considerable uncertainty and waste a lot of time.
I don't know how much I agree with the idea that noise should average out (it depends on the type of noise, maybe it could compound) or if optimizations cumulatively add up that easily. In my mind it only makes sense to benchmark a representative load using real world data and see what can be done there.
Hmm... Do you have an example of a type of noise where the average over many measurements compounds. I have never heard such a suggestion and cannot think of one.