[Solved] Run "cargo test" 1_000_000 times

#1
  1. I have been debugging some race conditions involving CUDA (unit tests can fail non-deterministically). So far, I’m doing something like:
while true; do cargo test --features="gpu"; done
  1. However, I’d prefer to have cargo test just run each test 1_000_000 times. Is there a way to do this?

Pre-emptive questions:

  • Why not put a for loop in the unit test itself?

Sometimes, I want to do a quick test of “is the algorithm even approximately correct” (and have it only run once). It’s only after I’m pretty sure the algorithm is “basically correct” what I want to run a torture test.

1 Like

#2

My preference for running the test 1_000_000 times within cargo test is due to all the extra output the while-loop method generates.

0 Likes

#3

I don’t think there is a way built into Cargo.

Add a feature to your Cargo.toml and then run the tests like cargo test --features="torture"?

#[test]
fn mytest() {
    assert_eq!(1 + 1, 2}
}

#[cfg(features=["torture"])]
#[test]
fn my_torture_test() {
    for _ in 1_000_000 {
        mytest()
    }
}
5 Likes

#4

I like this approach. Is there a way to have feature “torture” auto turn on feature “gpu” ? Or does rust’s 'additive feature" requirement prevent this?

0 Likes

#5

Sure can. Off the top of my head, you just have to put

#[features]
gpu = []
torture = ["gpu"]

in your Cargo.toml.

1 Like

#6

An alternative method is to call the test harness directly instead of using Cargo:

cargo clean # otherwise, you could end up with more than one target/test/YOUR_CRATE_NAME-* executable
cargo test --no-run --features="gpu"
while true; do `find target/debug -maxdepth 1 -name YOUR_CRATE_NAME-\* -type f -executable`; done
2 Likes

#7

Good idea, although the calling find at each iteration loop itches me:

#!/bin/sh
set -eu
set -o pipefail


export NUMRUNS=1000000  # or whatever amount you want
export CRATENAME=example # set to your crate name
cargo clean
cargo test --no-run # + whatever other flags you need, e.g. --features "gpu"
export TESTBINARY=$(find target/debug -maxdepth 1 -name $CRATENAME-\* -type f -executable)
seq 1 $NUMRUNS | while read line; do $TESTBINARY; done
1 Like