How do you debug a test with random input?

[culebron]

I have a test where I modify my data values (via the "official" API) by random deltas. I modify a handful of values, and sometimes one of them triggers an edge case (about 1 failure in 20-50 runs).

I log a lot of stuff, but logs are so numerous that they won't even fit in VSC screen, so that I can't click "run" and see what went wrong in the failed run.

This raises a question 1: what's the approach to silence the irrelevant debug prints?

I've added a call to save the inputs in a file, and when the test crashes, save them and then, probably, make another test that takes this fixed set of inputs, so that I can debug it.

Question 2: how do you make reproducible tests with random input?

[jdahlstrom]

Q2: Presumably your test inputs are not actually random, but pseudo-random? In which case repeatibility is just a matter of keeping the seed.

[akrauze]

Maybe you should try using proptest, fuzzing, or other method to iterate over input space to automatically check which inputs make your tests fail. Those tools usually have methods to produce minimal failing inputs, which can further help you with debugging.

[RedTechTiger]

Just to add my few cents, I usually use a chacha-based pseudorandom function that I seed with a known value, which generates reproducible results. Though fuzzing may be a better solution, depending on your needs.

Is all of this logged in a single test? If not, try running cargo test name-of-your-test. Though non-failing tests shouldn't print to stdout/stderr in either way, so this may be irrelevant. If you are using assert!(), consider using assert_eq!() or another appropriate comparison, as they log the values that causes the panic.

[newpavlov]

@culebron

what's the approach to silence the irrelevant debug prints?

Configure your logger to output debug level only for desired modules/crates (e.g. see the env_logger docs).

@culebron

how do you make reproducible tests with random input?

Create a PRNG instance (e.g. ChaCha8Rng) with a fixed seed and use it to generate test inputs.

[culebron]

@RedTechTiger

Is all of this logged in a single test? If not, try running cargo test name-of-your-test. Though non-failing tests shouldn't print to stdout/stderr in either way, so this may be irrelevant. If you are using assert!(), consider using assert_eq!() or another appropriate comparison, as they log the values that causes the panic.

Yes, it's all logged in one test, and logs are hard to browse. I do use assert with messages, but I can't understand how that value got wrong.