Intel's Variable Clock Speeds and Benchmarking

21

Yea, that has been the case, additionally, if you switch the order, the results will change too. Imo, benchmarking and profiling in Rust are not mature yet, especially for async code. There is a built in benchmark feature in nightly similar to Golang's bench, but because it is in nightly, it is not stable yet and looks like does not support async. The problem I encountered is that simply reversing the order of the written code, from Bench A then Bench B, to Bench B then Bench A, swapped the results, so where A was previously faster, B became faster instead. For serious benchmarking, consider getting a CPU that supports overclocking so the CPU frequency remains constant, setting it to a lower fixed speed to avoid high temperatures, and then warming up the cache by running the first few iterations without timing them

Since my CPU does not support overclocking, the first thing I do is read the assembly, which is the most deterministic way to identify performance differences. You can isolate the code into separate functions, annotate them with #[unsafe(no_mangle)], install cargo-asm, and run cargo asm --release. You can pass target cpu native to see full wide SIMD or potential other optimization. Or passing target CPU flags to cross compile, for example inspecting x86 64 assembly from ARM 64 (I did this from Termux). Even better, you can use Godbolt, as it will point out which specific line of code produced each line of assembly

From there, you can determine which parts can be optimized. For instance, you may frequently see the compiler fail to remove bounds checks because the values are evaluated at runtime, then you can apply the optimization like if you are certain the value will always stay within the bounds, you can use unsafe get_unchecked. You might also see that the code is not compiled into SIMD instructions, so you can edit it, like what I mostly do is removing branching code inside a loop until it stops generating jump instructions and vectorizes properly. If SIMD only utilizes half of the CPU's SIMD lanes, you can try passing target native or doing a manual unroll with fixed chunk sizes, though you should watch out because manual unroll can sometimes spill to the stack in different SIMD lanes (use the target flag to cross compile to inspect it) because stack is slower. Based on the assembly after all optimizations, you can also determine if now the execution becomes memory bound, then changing allocation strategy and allocator, trying memory prefetching, or configure NUMA binding. If you are ever confused by the assembly, simply paste it into AI tool and ask for a line by line breakdown, over time you will understand it naturally, which is how I learned

However, in a large codebase, you still need to profile it first to know which specific section of code requires assembly inspection. That is where I struggle, as I have not found an easy to use profiling tool for Rust, and existing tools produce graphs that are difficult for beginners to interpret. I think the tooling team and ecosystem should focus on creating easy to use profiling and benchmarking tools that yield easily understandable results, stay resilient against measurement noise, and support all async runtimes. Investing in these tools will ensure optimization actually produce expected results. Because bad profiling and benchmarking tools will misguide optimizations

The second is using Perf (Linux) metrics that are unaffected by CPU frequency scaling, such as branch misses, cache misses, CPU migrations, false sharing, etc

The third is, increasing the input data size and the total run so the workload is bigger and then the different can also be seen bigger, making it is easier to determine

22

The main one I could think of is the system throttling itself during the benchmark. This would be noise that is correlated together.

23

Well, that's how you run benchmarks on a Chromebook if you really need to do these on a Chromebook. No one was ever optimizing them for a way to run benchmarks on a device that's not a test device.

Yes, this is expected.

This wouldn't help you at all. Difference between cores of more than 1% is typical. That's why I wrote that you need to do core isolation, too. And ensure that benchmark runs on one specific core.

Yes. That's why having the test device is good idea: you lock everything and then producing results that can show you 0.5% improvemnt is feasible. But it's not feasible to also use that device for development. Because you are making it artificially slow to ensure that overheating is not interfering with benchmarking process.

Also: Chromebooks tend to be cheap, having separate one for benchmarking is not too expensive, thus no one was thinking about making it practical to run benchmarks on your work device.

They do add up pretty decently. At my $DAYJOB we have made our JIT-compiler two times faster over few years of development by doing small incremental 1-2% improvements. LLVM's improvements tend to be in the same region, too.

But to achieve cumulutive improvement that's noticeable not just on benchmarks you need a lot of such tiny improvements and years of development… compared to that price of one or dozen of Chromebooks is trivial.

Yes, that's true with modern development style where you replace 100 bytes of code with 100 megabytes of code because that saves you five minutes of development.

But not every project is at the stage where 99.99% of resources are wasted for “ease of development”, and even then you tend to have a runtime that's optimized well, “down below”, under all these wasteful layers of abstractions.

That's typical when you forget to benchmark on an isolated CPU core. Kernel takes about 100ms to ramp up the frequency, that has nothing to do with maturity of async benchmarking code.

24

Out of curiosity, have you considered using the number of cycles instead of the elapsed time? I think some benchmark tools let you see that, along with the number of cache and RAM accesses.

25

As can be seen in the README, unfortunately, there is a plenty of caveats.

26

I have tried pinning, the noise is there :<. I didn't refer it to async in my comment, but the nightly bench feature (additionally also criterion, manual timing). You can ensure consistent result in CPU that can lock to specific clock speed, or use assembly method like me, the assembly method is easier for me as nothing is hidden, I can see all the thing

But pure algorithm design (not low level component), will need benchmark first. Once getting the best design that I can, then my next step is squeezing more from looking the assembly

27

Oh, my mental picture of how the hardware works was wrong, I just found out cores do not all run off the same clock. Google search sayeth:

"A** multi-core Intel CPU has multiple individual clock signals**. While they usually share a single master reference clock generator on the chip, every core features its own local clock controls, dividers, and clock-gating circuits. This setup allows individual cores to run at different speeds or turn off entirely to save power".

So you may have 8 cores all running at different speeds at different times. Hmm, that seems a tough problem without being able to reach into the box and have some control.

Edit I did find this: sched_getcpu in rustix::thread - Rust .
But if the cores are independently varying their clock speeds, it seems a bit hopeless.

28

I don't give up easily. So I am thinking of another approach, try the test quite a few times, the result is the fastest time. Note which core the test ran on. So if it runs on different cores, and some are faster than others, well the fastest core wins. The same for various other variations that may be going on. Automate it so you can run a short test maybe 100 times or a 1,000 times (with some kind of recovery interval so the CPU doesn't get too hot) to get the best possible time. Then change the software and try again.

29

If you don't do core isolation then it may easily run on different cores during one run.

Indeed using fastest time is the best strategy.

But, ultimately, having a dedicated, properly prepared, test device before doing other optimizations reduces performance jitter by factor of 100… other tricks are much less important comparted to that.

30

Use core isolation and core pinning, plus of course disabling turbo and pstate. Preparation is key to benchmarking. If you refuse to do so it is a case of garbage in, garbage out.

But then you also need to be careful when analysing the results. Typically the average is not what you want, since the probability distribution of a benchmark is not normal. Rather, depending on what you are concerned about you may want to look at the min (how fast can this be if there isn't noise on top), the max (what is the worst case), the p95/p99 (what is the tail latency), the modes (is there something funny going on where the distribution is multi-modal), and so on...