HOWTO: Generating a branch coverage report


#1

Code coverage is quite useful if you want to know how good your tests are. Personally, I find line coverage not particularly insightful except for a very rough overview; on the other hand, statement coverage / condition coverage is invaluable in figuring out which tests are yet to be written. Unfortunately, condition coverage wasn’t available for Rust—until now!

See the full sample report.

Prerequisites

You will need:

  • an LLVM installation; specifically, the llvm-cov tool;
  • an LCOV installation; specifically, the lcov and genhtml tools;
  • a custom rustc build upstream rustc has profiling support

On a Debian derivative (e.g. Ubuntu) these can be installed with:

apt-get install llvm-3.9 lcov

To produce the custom rustc build, apply the compiler-rt buildsystem patch, then build rustc using rustbuild (the old buildsystem is not updated in the patch):

mkdir build
cd build
../configure --enable-rustbuild --prefix=~/rust-branch-coverage
make install
rustup toolchain link branch-coverage ~/rust-branch-coverage

Create a file llvm-gcov with the following contents and place it somewhere in your PATH:

#!/bin/sh -e
llvm-cov gcov $*

Preparing your crate

To gather useful branch coverage information, your crate must have tests. Further:

  • every test must pass, and
  • no test should cause a panic, including #[should_panic] or any other panic that is caught.

The reason for this is stated below.

First, update the test profile for your crate as follows to remove as many redundant branches as possible:

[profile.test]
opt-level = 1
codegen-units = 1

To build the unit tests, run the following command:

cargo rustc -- --test -C link-dead-code -Z profile -Z no-landing-pads

then manually run the target/debug/deps/$CRATE-$SUFFIX executable, in the crate root.

To build an integration test, use:

cargo rustc --test $TEST -- -C link-dead-code -Z profile -Z no-landing-pads

then manually run the target/debug/$TEST-$SUFFIX executable, in the crate root.

Running the cargo rustc command will generate a *.gcno file, and running the executable will generate a *.gcda file. Running an executable several times, say with different arguments, will merge the data from multiple runs.

Creating a coverage report

From the crate root, run:

LCOVOPTS="--gcov-tool llvm-gcov --rc lcov_branch_coverage=1"
LCOVOPTS="${LCOVOPTS} --rc lcov_excl_line=assert"
lcov ${LCOVOPTS} --capture --directory . --base-directory . \
  -o target/coverage/raw.lcov
lcov ${LCOVOPTS} --extract target/coverage/raw.lcov "$(pwd)/*" \
  -o target/coverage/raw_crate.lcov

This will display a short summary in your terminal:

Summary coverage rate:
  lines......: 78.0% (362 of 464 lines)
  functions..: 71.1% (59 of 83 functions)
  branches...: 45.7% (380 of 831 branches)

Then, run:

genhtml --branch-coverage --demangle-cpp --legend \
  -o target/coverage/ \
  target/coverage/raw_crate.lcov

This will generate an HTML report in target/coverage/.

Troubleshooting

Q: I get a lot of linker errors along the lines of undefined reference to `llvm_gcda_[something]’, what’s wrong?

A: You are not using the custom rustc, see the “Prerequisites” section.

Q: I get a lot of errors from lcov along the lines of cannot merge previous GCDA file: corrupt [something], what’s wrong?

A: You have ran an executable with branch coverage enabled, then rebuilt it, and ran again, thus attempting to merge the coverage information from two different compilations. Remove the *.gcda files and try again.

Q: Some of my code is not counted at all, what should I do?

A: Is this code in a generic function? Unfortunately you have to call it. Is this code in an #[inline(always)] function? Try changing that to #[inline].

Q: I have several test executables, how do I combine coverage from these?

A: Use the guide above to create separate files coverage_crate_a.info, coverage_crate_b.info, …, then combine them with:

lcov ${LCOVOPTS} -a coverage_crate_a.info -a coverage_crate_b.info ... \
    -o coverage_crate.info

Drawbacks and pitfalls

All in all, I feel that collection of branch coverage has achieved my goal—it is helping me write better tests. However, it is not fully general and there are a few issues:

  • Panicking is not supported. Technically, it is possible to build an executable with landing pads, but then every landing pad (in effect, every function call) will represent an additional branch. Having a branch on every Box::new, Option::unwrap_or and the like adds too much noise. (-Z no-landing-pads is used instead of -C panic=abort because the latter is ignored while building unit tests.)
  • Coverage for match is… bizarre. Often there will be far more branches than, say, irrefutable cases. -C opt-level=1 improves this but not by much. -C opt-level=2 doesn’t change anything.
  • Every arithmetic operation includes a branch that, in all likelihood, will never be taken. This adds a moderate amount of noise. Arguably, this is what should happen.

Possible improvements

  • The rustc patch should be integrated, see PR#38608. landed
  • Adding meaningful support for panicking is probably out of question. At least, I do not have any ideas. If you do, please share them!
  • Improving coverage for match is almost certainly possible, but requires some staring at the disassembly. I haven’t even tried. I may try if there is interest. Alternatively, simply adding all match statements into the exclusion regex and relying on the coverage for individual cases is also possible.
  • It’s not immediately obvious what to do with arithmetic overflow checks. One option is to build with debug-asserts = false to remove the check entirely.

#2

It is possible to generate the coverage data without waiting for #38608 or building a custom rustc, by linking to a pre-built profiler library. It even works with 1.17.0 stable. I’ve checked on both macOS and Debian.

The problem of skipping the proper fix #38608, AFAIK, are:

  • There might be incompatibility problem due to compiler_rt version difference
  • The coverage metadata is not stored into the debug info.

Instructions

  1. Install the necessary library

    • macOS: install Xcode command line tools.
    • Debian/Ubuntu: apt-get install libclang-common-3.8-dev (or apt-get install clang to pull it in via dependency) (higher LLVM version should also work)
  2. Clean up.

    cargo clean && rm -f *.gcda *.gcno
    
  3. Build the test. Only the last 2 lines (-L… -l…) are added to OP’s instruction.

    # macOS:
    cargo rustc -- --test \
                   -Ccodegen-units=1 \
                   -Clink-dead-code \
                   -Cpasses=insert-gcov-profiling \
                   -Zno-landing-pads \
                   -L/Library/Developer/CommandLineTools/usr/lib/clang/8.1.0/lib/darwin/ \
                   -lclang_rt.profile_osx
    # Debian 9.0
    cargo rustc -- --test \
                   -Ccodegen-units=1 \
                   -Clink-dead-code \
                   -Cpasses=insert-gcov-profiling \
                   -Zno-landing-pads \
                   -L/usr/lib/llvm-3.8/lib/clang/3.8.1/lib/linux/ \
                   -lclang_rt.profile-x86_64
    
  4. Verify a .gcno file is generated.

    ls *.gcno
    
  5. Run the test

    ./target/debug/deps/«name-of-the-executable»
    
  6. Verify a .gcda file is generated

    ls *.gcda
    
  7. Continue from OP’s “Creating a coverage report” step.


#3

I’m quite confused about the behaviour I see when using this to measure coverage. Here’s a reduced case that I expected to work, but doesn’t:

src/lib.rs:

  1 pub fn hello(who: &str) {
  2     println!("hello {}", who);
  3 }

examples/sample.rs:

  1 extern crate testcov;
  2 use testcov::hello;
  3 
  4 fn main() {
  5     hello("world");
  6 }

I expect that building this all, then running the sample executable will result in some coverage of the hello function. However, that doesn’t happen.

Steps I’m taking:

$ cargo rustc --profile dev --example sample -- -Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads -L/usr/lib/llvm-3.8/lib/clang/3.8.1/lib/linux/ -lclang_rt.profile-x86_64
   Compiling testcov v0.1.0 (file:///home/jbp/testcov)
warning: the option `Z` is unstable and should only be used on the nightly compiler, but it is currently accepted for backwards compatibility; this will soon change, see issue #31847 for more details


    Finished dev [unoptimized + debuginfo] target(s) in 0.28 secs
$ ./target/debug/examples/sample
hello world

$ ls -lha
...
-rw-r--r--   1 jbp jbp  152 May 28 16:13 sample.gcda
-rw-r--r--   1 jbp jbp  400 May 28 16:12 sample.gcno
...
$ lcov --gcov-tool ~/rustls/admin/llvm-gcov --rc lcov_branch_coverage=1 --rc lcov_excl_line=assert --capture --directory . --base-directory . -o cov.info
Capturing coverage data from .
Found LLVM gcov version 3.8.1, which emulates gcov version 4.2.0
Scanning . for .gcda files ...
Found 1 data files in .
Processing sample.gcda
Finished .info-file creation

$ lcov --gcov-tool ~/rustls/admin/llvm-gcov --rc lcov_branch_coverage=1 --rc lcov_excl_line=assert --extract cov.info "$PWD/*" -o cov-reduce.info
Reading tracefile cov.info
Extracting /home/jbp/testcov/examples/sample.rs
Extracted 1 files
Writing data to cov-reduce.info
Summary coverage rate:
  lines......: 100.0% (2 of 2 lines)
  functions..: 100.0% (1 of 1 function)
  branches...: no data found

$ genhtml --branch-coverage --demangle-cpp --legend cov-reduce.info -o target/coverage/ --ignore-errors source
Reading data file cov-reduce.info
Found 1 entries.
Found common filename prefix "/home/jbp/testcov"
Writing .css and .png files.
Generating output.
Processing file examples/sample.rs
Writing directory view page.
Overall coverage rate:
  lines......: 100.0% (2 of 2 lines)
  functions..: 100.0% (1 of 1 function)
  branches...: no data found

It looks like the actual crate contents doesn’t get instrumented for coverage for example programs?


#4

The problem in your example is that cargo rustc is actually going to invoke rustc twice. Try adding the --verbose option and you will see.

The documentation for cargo rustc indeed says:

The specified < opts >… will all be passed to the final compiler invocation, not any of the dependencies.

The solution is to use RUSTFLAGS instead, which passes the flags to all compiler invocations:

RUSTFLAGS="-Ccodegen-units=1 -Clink-dead-code -Clink-arg=-Wl,--whole-archive -Cpasses=insert-gcov-profiling -Zno-landing-pads -L/usr/lib/llvm-4.0/lib/clang/4.0.0/lib/linux/ -lclang_rt.profile-x86_64" cargo rustc --example sample

#5

Is there any way something like this could be turned into a cargo sub-command, so generating coverage is just a case of running cargo cov?


#6

Well, a cargo subcommand is just a program with a cargo-... filename in ~/.cargo/bin/ so getting a good, solid cargo subcommand would mainly consist of the following:

  1. Finish the process of merging the patches into rustc so no custom build is required. (issue 42524)
  2. Come up with a way to post-filter the lcov output (or reinvent lcov if you’re a masochist) so panicking can be supported without making the coverage report unusably noisy. (This would also be a good time to tidy up the output for match.)
  3. Decide on a solution for arithmetic overflow checks with broad appeal.
  4. Write a glorified shell script named cargo-cov which runs all of these commands for you and has comprehensive error checks (including for "<name of dependency> not installed") with good, easy-to-understand error messages.

#7

The rustc PR landed yesterday \o/ https://github.com/rust-lang/rust/pull/42433#issuecomment-308726834


#8

There’s already a Rust alternative to lcov at https://github.com/marco-c/grcov.


#9

Thanks. I’ve added it to my list of resources.


#10

Thanks, that was indeed the problem.

I now have full test coverage working for my crate, in 4 minutes rather than the 48 minutes that kcov took :smile:

One non-obvious thing I had to do was provide a RUSTC_WRAPPER that only provided the specific coverage options to rustc when compiling particular crates/outputs. Otherwise, your crate dependencies and their build scripts also get coverage, and the build fails when the second different build_script_build runs.


#11

Hi, I’m following -Z profile examples but got following errors:

./target/debug/deps/gcov_test-2a49a26c6bfb34c3

running 1 test
test tests::test_main ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x00000000)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x00000000)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x00000000)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous run count: corrupt object tag (0x00000000)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x61326635)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous run count: corrupt object tag (0x00004532)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x61326635)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous run count: corrupt object tag (0x00004532)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous GCDA file: corrupt arc tag (0x00000000)
profiling: /gcov-test/target/debug/deps/gcov_test-2a49a26c6bfb34c3.gcda: cannot merge previous run count: corrupt object tag (0x61326635)

There are one gcda and one gcno file generated.

However, when I use cargo rustc -- -Clink-dead-code -Ccodegen-units=1 -Zno-landing-pads -Cpasses=insert-gcov-profiling -L/Library/Developer/CommandLineTools/usr/lib/clang/9.1.0/lib/darwin/ -lclang_rt.profile_osx. There are several gcda and gcno files generated. And there is no error in building the code.

BTW, I’m using the latest nightly version:

$ rustup show
Default host: x86_64-apple-darwin

installed toolchains
--------------------

stable-x86_64-apple-darwin
nightly-x86_64-apple-darwin (default)

active toolchain
----------------

nightly-x86_64-apple-darwin (default)
rustc 1.27.0-nightly (ad610bed8 2018-04-11)