Code coverage is quite useful if you want to know how good your tests are. Personally, I find line coverage not particularly insightful except for a very rough overview; on the other hand, statement coverage / condition coverage is invaluable in figuring out which tests are yet to be written. Unfortunately, condition coverage wasn't available for Rust—until now!
See the full sample report.
Prerequisites
You will need:
- an LLVM installation; specifically, the
llvm-covtool; - an LCOV installation; specifically, the
lcovandgenhtmltools; -
a custom rustc buildupstream rustc has profiling support
On a Debian derivative (e.g. Ubuntu) these can be installed with:
apt-get install llvm-3.9 lcov
To produce the custom rustc build, apply the compiler-rt buildsystem patch, then build rustc using rustbuild (the old buildsystem is not updated in the patch):
mkdir build
cd build
../configure --enable-rustbuild --prefix=~/rust-branch-coverage
make install
rustup toolchain link branch-coverage ~/rust-branch-coverage
Create a file llvm-gcov with the following contents and place it somewhere in your PATH:
#!/bin/sh -e
llvm-cov gcov $*
Preparing your crate
To gather useful branch coverage information, your crate must have tests. Further:
- every test must pass, and
- no test should cause a panic, including
#[should_panic]or any other panic that is caught.
The reason for this is stated below.
First, update the test profile for your crate as follows to remove as many redundant branches as possible:
[profile.test]
opt-level = 1
codegen-units = 1
To build the unit tests, run the following command:
cargo rustc -- --test -C link-dead-code -Z profile -Z no-landing-pads
then manually run the target/debug/deps/$CRATE-$SUFFIX executable, in the crate root.
To build an integration test, use:
cargo rustc --test $TEST -- -C link-dead-code -Z profile -Z no-landing-pads
then manually run the target/debug/$TEST-$SUFFIX executable, in the crate root.
Running the cargo rustc command will generate a *.gcno file, and running the executable will generate a *.gcda file. Running an executable several times, say with different arguments, will merge the data from multiple runs.
Creating a coverage report
From the crate root, run:
LCOVOPTS="--gcov-tool llvm-gcov --rc lcov_branch_coverage=1"
LCOVOPTS="${LCOVOPTS} --rc lcov_excl_line=assert"
lcov ${LCOVOPTS} --capture --directory . --base-directory . \
-o target/coverage/raw.lcov
lcov ${LCOVOPTS} --extract target/coverage/raw.lcov "$(pwd)/*" \
-o target/coverage/raw_crate.lcov
This will display a short summary in your terminal:
Summary coverage rate:
lines......: 78.0% (362 of 464 lines)
functions..: 71.1% (59 of 83 functions)
branches...: 45.7% (380 of 831 branches)
Then, run:
genhtml --branch-coverage --demangle-cpp --legend \
-o target/coverage/ \
target/coverage/raw_crate.lcov
This will generate an HTML report in target/coverage/.
Troubleshooting
Q: I get a lot of linker errors along the lines of undefined reference to `llvm_gcda_[something]', what's wrong?
A: You are not using the custom rustc, see the "Prerequisites" section.
Q: I get a lot of errors from lcov along the lines of cannot merge previous GCDA file: corrupt [something], what's wrong?
A: You have ran an executable with branch coverage enabled, then rebuilt it, and ran again, thus attempting to merge the coverage information from two different compilations. Remove the *.gcda files and try again.
Q: Some of my code is not counted at all, what should I do?
A: Is this code in a generic function? Unfortunately you have to call it. Is this code in an #[inline(always)] function? Try changing that to #[inline].
Q: I have several test executables, how do I combine coverage from these?
A: Use the guide above to create separate files coverage_crate_a.info, coverage_crate_b.info, ..., then combine them with:
lcov ${LCOVOPTS} -a coverage_crate_a.info -a coverage_crate_b.info ... \
-o coverage_crate.info
Drawbacks and pitfalls
All in all, I feel that collection of branch coverage has achieved my goal—it is helping me write better tests. However, it is not fully general and there are a few issues:
- Panicking is not supported. Technically, it is possible to build an executable with landing pads, but then every landing pad (in effect, every function call) will represent an additional branch. Having a branch on every
Box::new,Option::unwrap_orand the like adds too much noise. (-Z no-landing-padsis used instead of-C panic=abortbecause the latter is ignored while building unit tests.) - Coverage for
matchis... bizarre. Often there will be far more branches than, say, irrefutable cases.-C opt-level=1improves this but not by much.-C opt-level=2doesn't change anything. - Every arithmetic operation includes a branch that, in all likelihood, will never be taken. This adds a moderate amount of noise. Arguably, this is what should happen.
Possible improvements
-
The rustc patch should be integrated, see PR#38608.landed - Adding meaningful support for panicking is probably out of question. At least, I do not have any ideas. If you do, please share them!
- Improving coverage for
matchis almost certainly possible, but requires some staring at the disassembly. I haven't even tried. I may try if there is interest. Alternatively, simply adding all match statements into the exclusion regex and relying on the coverage for individual cases is also possible. - It's not immediately obvious what to do with arithmetic overflow checks. One option is to build with
debug-asserts = falseto remove the check entirely.
