How to mark a test 'can fail sometimes'?

I am reading Testing - The Rust Reference but can't find a suitable attribute that allows a test to fail sometimes e.g., timeout error, remote server down, user permissions issues etc.

  • ignore means don't run the test, just compile it.
  • should_panic will expect test to fail all the time.

Something similar to pytest's xfail or skip_if (How to use skip and xfail to deal with tests that cannot succeed — pytest documentation)?

PS: I'd rather avoid something like https://nexte.st/ for this.

There aren't any standard ways to simply mark a test as flaky, unfortunately. You need to implement retries internally to the test or catch/handle errors which should not mark the test as failed.

Alternatively, you can mark all of the flaky tests as #[ignore] and use cargo test --include-ignored to, well, include the ignored tests. You can also use --ignored to just run the ignored tests. This only really works if all of your #[ignore]d tests are supposed to be able to be run in this manner.

2 Likes