Test Failure message of ```should_fail``` and ```expected``` works for panic! but not for assert_eq!


#1

According to a Rust documentation, to prevent tests failing for unexpected reasons it is suggested that we adopt a safer test harness by accompanying the should_fail attribute with an optional expected parameter, as they say this will make sure the failure message contains the provided text.

  • When panic! is used with the following code, it only partly works as expected when I run cargo test by displaying the following under the first failure: heading: ---- it_fails_equality_comparison stdout ---- thread 'it_fails_equality_comparison' panicked at 'PLEASE PANIC WHEN PANIC! USED', src/lib.rs:54, however I expected it to also display the other message (i.e. PLEASE PANIC WHEN EXPECTED USED) under the failure message stdout heading, which it doesn’t (see extract from terminal failure message below)
#[test]
#[should_fail(expected = "PLEASE PANIC WHEN EXPECTED USED")]
fn it_fails_equality_comparison() {
  panic!("PLEASE PANIC WHEN PANIC! USED");
}
running 1 test
test it_fails_equality_comparison ... FAILED
failures:
---- it_fails_equality_comparison stdout ----
	thread 'it_fails_equality_comparison' panicked at 'PLEASE PANIC WHEN PANIC! USED', src/lib.rs:54
failures:
    it_fails_equality_comparison
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
thread '<main>' panicked at 'Some tests failed', /Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/src/libtest/lib.rs:256
  • When instead we use assert_eq! in the following code, according to the documentation this should display PLEASE PANIC WHEN EXPECTED USED in the failure message stdout, but it doesn’t (see extract from terminal failure message below)
#[test]
#[should_fail(expected = "PLEASE PANIC WHEN EXPECTED USED")]
fn it_fails_equality_comparison() {
  assert_eq!("english", "english");
}
running 1 test
test it_fails_equality_comparison ... FAILED
failures:
failures:
    it_fails_equality_comparison
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
thread '<main>' panicked at 'Some tests failed', /Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/src/libtest/lib.rs:256
  • Strangely, when I run the following code, I expect it to pass, but it fails… It appears to ignore the fact that I have applied the should_fail attribute, and whilst I’m upset that it’s still not using the custom message defined with the optional expected parameter, it does otherwise give a logical and meaningful explanation of why it failed, but again, I don’t understand why it fails since we’re using should_fail
#[test]
#[should_fail(expected = "PLEASE PANIC WHEN EXPECTED USED")]
fn it_fails_equality_comparison() {
  assert_eq!("english", "german");
}
running 1 test
test it_fails_equality_comparison ... FAILED
failures:
---- tests::it_fails_equality_comparison stdout ----
	thread 'it_fails_equality_comparison' panicked at 'assertion failed: `(left == right) && (right == left)` (left: `"english"`, right: `"german"`)', src/lib.rs:53
failures:
    it_fails_equality_comparison
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
thread '<main>' panicked at 'Some tests failed', /Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/src/libtest/lib.rs:256
  • Lastly, if I change my code to the following, it yet again doesn’t display PLEASE PANIC WHEN EXPECTED USED in the failure message stdout (see extract from terminal failure message below), and in the place where I’d expect this to be displayed, it instead displays a different message assertion failed: false
#[test]
#[should_fail(expected = "PLEASE PANIC WHEN EXPECTED USED")]
fn it_fails() {
  assert!(false);
}
running 1 test
test it_fails ... FAILED
failures:
---- it_fails stdout ----
	thread 'it_fails' panicked at 'assertion failed: false', src/lib.rs:52
failures:
    it_fails
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
thread '<main>' panicked at 'Some tests failed', /Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/src/libtest/lib.rs:256

Why doesn’t the optional expected parameter work by displaying the custom message in the failure message stdout?

Why isn’t the should_fail attribute being interpreted correctly in all cases?


#2

The expected attribute means the test case passes only if the function panics with the exact message it is written in. This is not a mean to customize the output message.

#[test]
#[should_fail(expected="foo")]  // <-- these two should match
fn should_pass() {
    panic!("foo");              // <-- to pass the test case
}

#[test]
#[should_fail(expected="bar")]
fn should_fail() {
    panic!("foo");
}
running 2 tests
test should_pass ... ok
test should_fail ... FAILED
...

#3

@kennytm Even in this case, it would still be good to say something like Expected <> panic, got <> panic if the panic contains the wrong message.

Something like:

running 1 test
test it_fails_equality_comparison ... FAILED
failures:
---- it_fails_equality_comparison stdout ----
	thread 'it_fails_equality_comparison' panicked at 'PLEASE PANIC WHEN PANIC! USED', src/lib.rs:54
        expected 'PLEASE PANIC WHEN EXPECTED USED!', found 'PLEASE PANIC WHEN PANIC! USED'
failures:
    it_fails_equality_comparison
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
thread '<main>' panicked at 'Some tests failed', /Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/src/libtest/lib.rs:256

I think this is what @ltfschoen is asking for, but I may be misinterpreting his post. In any case, I think having the expected and found panic message in output would be helpful.