For our CI (as well as local build command), I'd like to be able to include doc tests into standard compilation of our project.
I have the feeling that doc-tests are not a 'usual' target selectable for e.g. cargo check or cargo build
Also cargo doc doesn't seem to include neither check nor build nor running the doc-test/example.
I only know of cargo test command being aware of the doc-test/examples in a way to check, build or run them.
I learned that cargo test indeed has both a) target selection for doc-tests/examples (parameter --doc) and b) option --no-run. But using both in combination I am shown this output error: Can't skip running doc tests with --no-run
What can I do to check/build all my doc-test/examples without running them (and other tests) at the same time?
BTW: I find terminology a bit confusing here: Examples in documentation sometimes seem to be referred to as: target: doc, doc-tests, examples, however this is not what examples mean in that sense, I suppose: Cargo target: examples as of Cargo Targets - The Cargo Book
As you have found, --doc is not really a "target" but just a flag. This means some of the options don't apply to it (e.g., --no-run). The only way I know to not run doc tests is by using the no_run attribute on the doc tests. This of course means you'll never run the tests as opposed to running the tests unless overridden via the --no-run flag. Something like:
As far as the terminology goes, yeah it's inevitable that terms will eventually become overloaded so hopefully it's "obvious" from context. Here testing "examples" means code in the examples/ directory that have the #[test] attribute.
Some other notes:
--doconly works for library crates, and you'll get an error if you use it in a package that doesn't contain a library crate.
--all-targets is almost equivalent to --benches --bins --examples --lib --tests with the only differences being an error will not occur when run on a package that doesn't have a library crate or a warning from being written to stderr in the case the other targets don't have a match.
As you noticed, cargo test is neither a superset nor subset of cargo test --all-targets as the former calls cargo test --doc while the latter does not but the latter tests targets (e.g., --examples) that aren't by the former.
cargo doc only builds the documentation. Essentially it only cares that it's valid Markdown and doesn't attempt to parse code blocks.
What does this all mean? The "safest" choice for maximal coverage is running cargo check/clippy --all-targets, cargo test --all-targets, cargo test --doc, and cargo doc. You may want to do all of those for each combination of "features" for both the stable and MSRV toolchains (if one is defined). Last, you'd do that on as many Rust targets as you can (e.g., x86_64-unknown-linux-gnu). In practice most crate authors only run some of those commands for some combinations of features for only the stable toolchain and only on a small number of Rust targets.
When it comes to cargo doc, if you only care that documentation will be built on docs.rs; then you may want to consider running RUSTDOCFLAGS="--cfg docsrs" cargo +nightly doc instead with any additional arguments specified in [package.metadata.docs.rs] (e.g., --all-features) in Cargo.toml. This is because docs.rs always uses the nightly compiler to build docs and always passes --cfg docsrs even if you haven't directed it to in [package.metadata.docs.rs]. One benefit of this is it allows you to conditionally enable the unstable doc_cfg feature based on docsrs so that your documentation will get "pretty" annotations when certain things are enabled (e.g., features) while still allowing the stable compiler to build your crates. Do note that you may want to also pass --no-deps when testing locally to both speed up the build but also to ignore any errors that may occur from docs in dependencies.
If your goal is to compile the doc-tests for the sake of build caching, then, this will not work: doc-tests are (unfortunately) always compiled each time they are run.
If your goal is to not run the doc-tests because they have undesirable side effects, I would recommend changing the tests so that they do not have those side effects.
(If your goal is to not run the tests because they are too slow for this phase of CI, then I have no recommendations.)
Part of the confusion here may be that rustdoc also (unstably) supports scraped examples, where code from example targets (examples/*.rs) is automatically displayed in the documentation of functions appearing in those examples. Those snippets are not doctests; the example target may be tested but if so, it is tested separately as a normal Cargo target and not as a doctest.
Other than that, it is true that code blocks written in documentation “are examples” in a documentation-writing sense (they appear in an # Examples section and readers of the documentation understand them as examples) and also “are doctests” (they are executed by rustdoc as tests). These are two different purposes that the same text serves.
Thank you @philomathic_life and @kpreid for your insights. What I learned from your answers basically corroborates what I assumed.
Here a bit more background why I came up with that. We are using VS Code as local development environment and a GitHub repo, which runs our CI for pull requests.
We have a cargo workspace with multiple crates, most of them have lib.rs, one is a bin-crate only.
We configured a default build task (e.g. run when Ctrl+Shift+B and also being the preLaunchTask - all this is VS Code terminology only, I suppose). This build task is run on workspace level. The build task itself is actually a a multi-step build task (For us, it includes cargo fmt, cargo check, cargo build --all-targets, cargo clippy). Intention of this build task is to give early, feedback as quickly as possible if there are any compilation or lint issues. We usually require it to succeed before we even run tests locally. So basically our mini-iteration cycle for local development works like this: change code -> build -> change code -> build -> change code -> build -> run tests -> change code -> .....
Now the desire was to include compilation of the doc-tests into the 'build' step, however because we considered running doc-tests rather to belong to the other 'run tests' step we don't also want them to be run necessarily in the build step.
Now I learned, there is no way to achieve that. So we compromised and added "cargo test --doc" into our build step and accept that it not only compiles, but also runs our doc-tests (some of them are using no_run anyways).