Cargo test --example with custom test harness

help
1

I've been looking into custom test harnesses like libtest-mimic, which implement the protocol needed for libtest and nextest. But all the uses I can find either use harness = false to make integration tests (tests/*.rs) have use their own main that hands off to libtest-mimic or they just use main() in examples.

I want examples that can run directly like cargo run --example fun_stuff --fun --args, but are also unit-testable via cargo test --example fun_stuff (or as a batch with cargo test --examples). With standard libtest, one can drop this at the end of examples/fun_stuff.rs to ensure it gets run that way:

#[cfg(test)]
mod tests {
    #[test]
    fn test_main() {
        super::main()
    }
}

(Of course you can also have ordinary unit tests.) This causes creation of a special testing executable and populates it with a testing main() that gets called instead of the ordinary main() in the example. I'd like to do something related to the above using a custom test harness, but I'm having trouble tracking down where this testing main() is defined and how to do that with a custom harness.

This is part of a broader goal to have example unit tests execute the example itself (passing through main()) with special per-test overrides and environment.

Any tips about how to find where the testing main() is defined and if/how I can hook in a custom harness?

2

I don't know if there is a way to employ the magic implicit main() replacement mechanism that the built-in harness uses (I doubt it), but you can mimic its effects while using harness = false by changing what main() does under cfg(test):

#[cfg(not(test))]
fn main() {
    todo!("normal non-test behavior goes here");
}

#[cfg(test)]
fn main() {
    todo!("start custom test harness here");
}
1 Like
3

Thanks. Since I want an entry point to the original also in test mode, it sounds like I'll need something like

fn ordinary_main() {
    todo!("normal logic goes here");
}

#[cfg(not(test))]
fn main() {
    ordinary_main();
}

#[cfg(test)]
fn main() {
    let ordinary_mode: bool = todo!("determine if test executable is run in this mode");
    if ordinary_mode { ordinary_main(); }
    else {
        todo!("rest of the custom test harness here");
    }
}

I was concerned this would be too confusing for users, but now I think I can make an attribute macro so the user writes something like this.

#[harness::example_main]
fn main() {
    todo!("normal logic goes here");
}
4

I'm not sure what exactly you are intending here. In the normal harness, there is no such thing; the test binary can't do anything but run its tests. Can you explain under what conditions the ordinary_mode would be needed? I suspect it is not.

5

I want each test in cargo test --example this to invoke

$ mpiexec -n 3 target/debug/examples/this-ff5c9134bbd067e4 --entry-point ordinary_main -- --various-options --for-ordinary-main

and analyze the output. That mpiexec -n 3 runs as a parallel 3-process job that communicates between the processes. I can't invoke the "normal" target/debug/examples/this because it doesn't get built by cargo test.

Outside this context, I don't love that examples are typically not run automatically.

6

I see. Cargo does support providing runnable binaries to tests (via the CARGO_BIN_EXE_* mechanism), but they have to be bin targets (not example or test targets).

For your particular use case of an example that needs to start itself to test itself, adding a flag to your custom test harness which starts the ordinary main does sound like the best option available.

Outside this context, I don't love that examples are typically not run automatically.

The purpose of example targets is to provide educational code that the user could run, but isn't necessarily suitable as an automated test — for example, it might open a window or otherwise expect user interaction. What would you want to happen instead, without making that difficult?

7

I think it's pretty common to test libraries using command-line examples that can be a starting place for a user and also provide a way to explore library functionality interactively. But in normal use, nothing automated tests that those examples actually run, and I've not-infrequently found projects with stale examples despite CI checking cargo test.

8

Oh, one last question: How does one specify that an example (as in cargo test --example x) uses a custom test harness? The syntax I'm familiar with is for integration tests.

9

That's a good point. You could propose and implement a feature to support this. (The way I'd do it is not put tests in the examples itself, because the example code should be free of distractions, but extend the CARGO_BIN_EXE_* mechanism to optionally cover examples as well, so that separate tests in tests/ could run the examples.)

harness = false applies to all kinds of targets, not just integration tests.

[[example]]
name = "x"
harness = false
10

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.