Why are doctests exempted from conditional compilation for tests?

help
1

Why are doctests exempted from conditional compilation for tests? :thinking:

Let's look at the following code. It defines a single constant ARGON2_MEMORY_COST using conditional compilation: If we're testing, then set it small, otherwise set it large. Below that constant, we have both a unit test, as well as a doctest. Both types of tests have assert_eq!(ARGON2_MEMORY_COST, 1024). However, only the unit test passes. The doctest has somehow been compiled under the non-test code, as we will later see.

#![allow(dead_code)]

#[cfg(test)]
pub const ARGON2_MEMORY_COST: u32 = 1024; // smaller for faster tests
#[cfg(not(test))]
pub const ARGON2_MEMORY_COST: u32 = 65536; // larger for enhanced security

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn memory_cost_is_small() {
        assert_eq!(ARGON2_MEMORY_COST, 1024);
    }
}

/// Placeholder struct just so we can show an example of a doctest.
/// 
/// # Examples
/// ```rust
/// use doctests_conditional_compilation::ARGON2_MEMORY_COST;
/// 
/// assert_eq!(ARGON2_MEMORY_COST, 1024);
/// ```
struct UnitStruct;

This is the output for the unit test:

running 1 test
test tests::memory_cost_is_small ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

On the other hand, this is the output for the doctest:

   Doc-tests doctests_conditional_compilation

running 1 test
test src/lib.rs - UnitStruct (line 20) ... FAILED

failures:

---- src/lib.rs - UnitStruct (line 20) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' panicked at /tmp/rustdoctestCdCfPy/doctest_bundle_2024.rs:8:1:
assertion `left == right` failed
  left: 65536
 right: 1024
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    src/lib.rs - UnitStruct (line 20)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

So, why are doctests excluded like this from conditional compilation which is explicitly aimed at test cases? And how can I configure the conditional compilation to include doctests in my testing code?

Any insight would be appreciated. :crab:

2

In general, cfg(test) is only set for a crate that is being compiled into a test executable that runs the #[test] functions directly in that crate.

Doctests[1] are compiled as separate crates which depend on your library crate compiled into a normal library, not a test executable. This might change in the future, but currently it is the only option.[2]

In case it is any consolation, a significant advantage of this architecture is that it means that your doctests accurately exercise the use of your library by dependents — if doctests were part of the same crate, then they could call private items entirely accidentally, failing to serve their role of being examples that are tested to work.

In my opinion, you should think about doctests as primarily being tested examples — code that a user of your library can copy into their own project and expect to succeed — not as tests that are also documentation.

  1. and test targets a.k.a. “integration” tests, too ↩︎

  2. There is a #[cfg(doctest)], but that relates only to collecting doctest code — finding items that have doctests in their documentation, not to compiling and executing it. ↩︎

3 Likes
3

For my fast & cheap inline strings :thread: Stringlet v0.3.0 I just came up with a new solution. In hindsight it’s so obvious, I wonder why it’s not the standard way. Sadly rust-analyzer still greys it out, unlike #[cfg(test)], which it always treats as active:

#[cfg(doctest)]
mod doctests {
    /**
    ```compile_fail
    let _x: stringlet::Stringlet<65>;
    ```
    */
    fn test_stringlet_65_compile_fail() {} // max SIZE 64 at compile time
}

Your take away is that you could work with #[cfg(any(test, doctest))].

#[cfg(doctest)] not set?