Rust does not appear to "fail fast" for individual tests

I looked up some issues in GitHub, and it appears it is the intent of Rust for cargo test to "fail fast". My understanding of "fail fast" is that tests are run until the first test fails, and the rest are ignored. This is what I've come to expect with test suites in JavaScript or Ruby.

By default, cargo test --no-fail-fast is available, but there is no --fail-fast flag available as this is the default.

I don't know if this behavior has changed since earlier versions, but I am using Rust 1.79.0.

My actual question is, why is there no test-level fail fast? It seems odd and arbitrary to fail fast at the module level rather than the individual test level. Is this intended? Is a more granular fail fast planned to be implemented?

This is my opinion, so take it with a grain of salt: I think the behavior should be by default no-fail-fast, and passing a --fail-fast flag will run the tests individually until one fails (like in many other languages' test suites). I think when people want to explicitly enable fail fast, they probably want individual test fail fast, and those who want to see an entire module's worth of tests typically would prefer to see all failures in all modules (for cd/ci). I struggle to see the use case a module-level fail fast has.

To avoid A/B issue: the reason I want this specific fail fast feature is because many rust learning courses use tests to break down the problem into individual steps. Being able to run tests sequentially and failing on the first test is useful and makes TDD easier in some cases.

I am open to critiques or explanations.

1 Like

Nitpick: the boundary is at the crate (or, equivalently, Cargo target) level, not module level. That's the level at which success/failure control flow is handled by the individual built test-binary (usually using rustc's built-in test harness, libtest), rather than Cargo's own code.

As to why not fail-fast: Tests in a given test binary are (by default) run in parallel, so if it stopped on the first observed failure, then which failure is reported would be nondeterministic, which is very bad for comprehension and the edit-then-retest workflow. So, it is important that these tests are no-fail-fast.

Also, my personal experience is that many bugs are hard to comprehend when looking only at a single test's assertion failure; the failure pattern and messages of multiple test functions are important to get a good perspective on where the bug is.

However, I don't have a particularly good argument for why Cargo shouldn't be no-fail-fast overall, and be consistent; only the broad intuitive expectation that if a crate has a bug caught by its test suite, there's much less likely to be value in also testing its dependents. (Of course, Cargo's behavior isn't actually to skip dependents of failures, it's to stop on the first failure.)

3 Likes

I agree with your perspective mostly. The only thing I have to point out is that if fail-fast was off by default, it wouldn't be to hard to run everything in one thread by locking --test-threads to 1. Everything is deterministic when set to 1 as far as I can see (I actually do this with TDD).

Sure, but then tests would take much longer to run, so it's a bad default. (I think test binaries should be run in parallel too.) Most runs of most tests pass.

1 Like

I agree

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.