Get number of iterations of a benchmark


#1

Is there any way to get the number of iterations that a benchmark will be run for? I need to pre-allocate a vector with one element per iteration. Currently, I’m just allocating “enough” and then panicking if it turns out not to be enough. Up the definition of “enough,” recompile, rinse, repeat. This is obviously a total hack.

fn bench_free_in_order(b: &mut Bencher) {
    let mut alloc = SlabAllocator::<T>::new();
    let mut ptrs = vec![];
    for _ in 0..100000000 {
        ptrs.push(alloc.alloc());
    }
    b.iter(|| alloc.free(ptrs.pop().unwrap()));
}

#2

Right now it doesn’t target a number of iterations but a total time to run. It will run it once and use that to figure out how many times to run it. Since this is internal implementation it’s subject to change.


#3

Ah, oh well. Is there some way to discount certain computation from the time counted towards a given run of the benchmark function? I’m thinking that I could just dynamically resize the vector if need be, but I’d want the time taken to do that not to count towards the benchmark time.

For example, in Go, this would be:

func BenchmarkFoo(b *testing.B) {
    for i := 0; i < b.N; i++ {
        b.Stop()
        // expensive stuff that shouldn't be counted
        b.Start()
    }
}

#4

Isn’t there a Bencher::bench_n fn that allows specifying # of iterations?


#5

Not in nightly unfortunately :confused:


#6

Hmm, ok - too bad. I might be misremembering such a thing existing then (or perhaps it’s not available yet).


#7

You’re not misremembering - it’s in stable; it just seems to have been removed from nightly.


#8

Ah yes, thanks. I’m on my phone so can’t quite dig around, but perhaps someone knows why it’s removed (hopefully because there’s a better replacement?).


#9

Not sure, but poking around the code it appears that there’s a full refactor of that code underway.


#10

The link to the bench_n method is from Rust 1.1 docs, which is very oldTM in rust terms.


#11

Ah, right you are. I’m used to Go where the only references available are for the current version, so I didn’t bother checking what version the Google result was.


#12

@bluss, do you happen to know why it was removed and if anything is planned to replace that functionality?


#13

I think the benchmarking code has been refactored a bit due to pending stabilization, but it’s unclear if that’s going to happen.

I’d encourage people to explore writing benchmarks runners for stable Rust, it’s entirely possible, and it would inform what kind of metaprogramming features are missing in Rust to enable it to be as smooth as the current #[bench] feature, yet with more features.