Weird benchmark differences caused by adding unrelated code

help
1

While benchmarking some code, I noticed that the performance of one benchmark was being affected by the other ones (when I commented out the other benchmarks, the first one became ~2x faster). I've minimized the code down to:

#![feature(test)]

extern crate test;

use test::Bencher;

#[bench]
fn bench(b: &mut Bencher) {
	b.iter(|| {
		[].is_ascii();
	});
}

/*// `#[no_mangle]` to prevent this from being optimized out
#[no_mangle]
fn breaks_it(b: &mut Bencher) {
	let bytes: &[u8] = &[];
	test::bench::iter(&mut || {
		bytes.is_ascii();
		Vec::new().resize(0, ());
	});
	b.iter(|| {});
}*/

Running cargo bench on this gives a time of 0.23 ns/iter for bench. If you uncomment the function breaks_it, the time goes up to 1.60 ns/iter.

The time goes back to 0.23 ns/iter if you comment any part of breaks_it, so it seems that the performance difference only happens when breaks_it contains:

  • 2 calls to test::bench::iter, at least 1 directly, others can be through Bencher::iter.
  • At least 1 use of b, (apparently black_box(b) doesn't count, so it must be either b.iter() or b.bench())
  • In the closures passed to test::bench::iter, there's a call to Vec::resize, and a call to [u8]::is_ascii on a slice borrowed from outside the closure.

The time also goes down to 0.23 ns/iter if I use the same closure for both calls to test::bench::iter, i.e.:

#[no_mangle]
fn breaks_it(b: &mut Bencher) {
	let bytes: &[u8] = &[];
	let mut inner = || {
		bytes.is_ascii();
		Vec::new().resize(0, ());
	};
	test::bench::iter(&mut inner);
	b.iter(inner);
}

The time also goes back down if I wrap both closures in breaks_it or the closure in bench in Boxes, but goes back up if I wrap all of them.

The time also goes down if I inline any of the functions used (test::bench::iter, [u8]::is_ascii, Vec::resize), so perhaps them being in the standard library changes something.

Incidentally, 0.23 ns/iter is the same time I get benching an empty closure, so it looks like in that case, is_ascii is being optimized out.

This implies that there's something about these very specific conditions that prevents optimizations in a different function.

Does anyone know what could cause this spooky action?

My rust version is 1.89.0-nightly (64a124607 2025-05-30).

2

it might just be the side effect of some unpredictable kernel or hardware states, such as code page alignment, cache or TLB collision/thrashing, etc.

this just reminds me of the "stablizer" paper.

1 Like
3

I suspect use of literal [] makes is_ascii possible to optimize out entirely, and then you're seeing different levels of inlining, which allows deleting more or less of the benchmark harness around the empty no-op closure.

4

From looking at the assembly, it seems rust will only inline [u8]::is_ascii up to 4 times, and the bench function emits 4 calls to is_ascii, so having another call elsewhere brought it over the limit. I have a revised MRE:

#![feature(test)]

extern crate test;

use std::hint::black_box;

use test::Bencher;

#[bench]
fn bench(b: &mut Bencher) {
	b.iter(|| {
		[].is_ascii();
	});
}

#[no_mangle]
fn breaks_it() {
	black_box::<&[_]>(&[]).is_ascii();
}

Interestingly, if I wrap the [] in bench in a black_box, it prevents the compiler from inlining the closure as much, meaning there are only two calls to is_ascii, meaning that can be inlined, bringing the time down to 0.23 ns/iter. (Adding black_box causes more inlining‽).

Also, even though [u8]::is_ascii is marked as const, [].is_ascii() isn't evaluated at compile time unless it ends up being inlined or I wrap it in const {}. Is this meant to happen? Wouldn't that mean any const function used 'too many' times will never be evaluated at compile time unless you explicitly tell it to?

5

A const fn doesn't imply that the function will necessarily be evaluated at compilation time. The only way to actually ensure a const fn is evaluated at compilation time is by invoking it in a const context (e.g., explicitly assign the value to a const variable). So yes, this is meant to happen. See The Reference for more information.

1 Like