Compile_fail doc test ignored in cfg(test)

help
1

I noticed that a few compile_fail I had in my unit tests were completely ignored when I was running cargo test.

I would like to make sure that the lifetimes of some borrows are seen correctly by the compiler, so I want to rely on those doc tests.

Here's what the test looks like when it's simplified for the sake of clarity:

#[cfg(test)]
mod tests {
    /// ```rust,compile_fail
    /// let value = "a".to_string();
    /// let borrow = &value;
    /// //let borrow_mut = &mut value; // commented out, so it should compile
    /// println!("{borrow}");
    /// ```
    fn must_not_compile() {}
}

If I run cargo test --doc, or cargo test, it's not even mentioned. If I force the test with cargo test --doc tests::must_not_compile, it shows 0 tests have been run.

For this to work, I must apparently remove the #[cfg(test)] directive. Then, any of the commands above reports a failing test.

I see a few issues:

  • Isn't that strange and misleading that a test is ignored when it's run in the test configuration section?
  • I actually keep my unit tests in a separate tests.rs file, to avoid cluttering the source code with them and, at the same time, to make it easier to see the tests and the code being tested next to each other in the editor - jumping from one to the other all the time in a single file is annoying and error-prone. Because of the top #![cfg(test)], I can't put those compilation tests in that file, unless I put individual #[cfg(test)] everywhere (not practical).
  • It requires to create a fake function in the source, must_not_compile, which is cumbersome to say the least. The alternative is to put the test in the doc of a "real" function, type, trait, ... , but that unnecessarily clutters the documentation.

Is there a better solution that I missed? I don't want to use crates like trybuild because it requires one separate Rust file per test (I think), and I have a bunch of them.

For now, I'm putting those tests in a separate file, which isn't cfg(test). It's the least awkward solution I found.

And is there a good reason not to run doc tests in a test configuration section?

2

As you probably realize, doctests are definitely not meant for use cases other than, well, attaching runnable/auto-validated examples to documentation, and the real issue you're having is that Rust/Cargo/the test runner doesn't currently permit writing must-not-compile tests except as doctests. And the reason for this is Rust's compilation model. Each doctest is compiled separately, as its own crate, so it's not difficult to support compile_fail with them. On the other hand, a crate containing many tests must either compile fully or not at all. Even if you had a separate crate where you isolate your must-not-compile tests, it would be very difficult to ensure that all the test functions fail to compile, not just one of them. Essentially, it must be one crate per test.

5 Likes
3

It’s not a specifically desired feature, but a consequence of how the whole system works. Doc-tests are extracted from documentation, and documentation is extracted from your crate compiled (well, parsed) without cfg(test) active.[1] Extracting cfg(test) doc-tests would require a completely separate run of rustdoc,[2] and that’s not a priority given that the purpose of doc-tests is to test documentation examples, not to provide a more general test facility. (incorrect, see below)

  1. And doc-tests are linked against your library compiled without cfg(test), because in the test mode, the library isn’t a library, but a test executable — but that’s irrelevant to this particular matter. ↩︎

  2. Because cfg conditions can be both positive and negative, and be used to choose between two different definitions of the same name, it’s not possible to just take both. ↩︎

1 Like
4

I see, thanks! I clearly had the wrong model of compilation process in my mind, thinking the test configuration included both the test and the non-test parts.

Admittedly, a fail-to-compile test is not as common in the history of unit testing as the other types of test. I'm only including it because if someone modifies the wrong lifetime or fails to include it in a function definition, the borrow checker could be unaware of some problems (in the specific case of the library I'm testing).

I'll keep the "compile_fail" tests in a separate file for the time being; I don't suppose it's a burden in the library because the empty functions below the doc tests are trimmed by the optimization, but a cleaner option in the long term would likely be one of those testing crates (e.g. trybuild), even if it's a little more tedious.

5

Well, in most cases it does, but Cargo doesn't assume that's true. More importantly, cfg(test) is expected to be only set when building a test binary — the thing that runs #[test] functions when it is run. That's not what rustdoc does, so if rustdoc set the same cfg, things might be inconsistent.

...and I just remembered that, actually, there is a feature to do exactly what you want, defining hidden tests: #[cfg(doctest)].

2 Likes
6

You don't need any functions for doc tests. Just put them in the documentation that's independent from any function (//! instead of ///).

7

You can also define documentation for an extern block:

/// things here
/// ```
/// assert!(true);
/// ```
extern {}
1 Like
8

I've seen that in the mean time, but I haven't tested it. Thanks for the tip!

EDIT: Changed to cfg(doctest); I prefer it that way.

Of course, one must not forget to run the tests in +nightly because the error codes aren't tested in stable. Quite error-prone, that.

9

Yes, I've tried that, but there's no point in cluttering the public documentation with test code. It might be interesting in some cases, to show what not to do, but not here.