Tutorial: How to collect test coverages for Rust project


I recently had to collect test coverages for my rust-strconv (and many others). It was not very easy and I had to do some yak shaving to get it working, but as I’ve done shaving you don’t have to!

Important warning: This tutorial would only work in x86 and x86_64 Linux. I don’t think it works in OS X and BSD either. (I do think that it is relatively easy to get kcov work in some BSDs like FreeBSD, but I don’t have any BSD box.)

Compiling kcov

We will use kcov to get the code coverage. Unfortunately, the old versions (including virtually all packaged versions) of kcov cannot handle the executable from Rust. You have to compile it manually.

You first need to get the basic dependencies: libcurl, libelf and libdw (but no libdwarf). I’ve confirmed the following command is enough for Ubuntu, use corresponding commands for other Linux distros.

$ apt-get install libcurl4-openssl-dev libelf-dev libdw-dev cmake gcc

Now get the kcov and compile.

$ wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz
$ tar xzf master.tar.gz
$ cd kcov-master
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install

This would install kcov into /usr/local/bin by default. You can customize the install path by using cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr .. or so.

If you happen to avoid make install (like me), you should copy both kcov and *.so files from the build/src directory. Shared objects are necessary to work with Rust executables.

Collecting coverage

Once you’ve got a fresh kcov in the PATH, you can collect the coverage:

$ cargo test --no-run
$ kcov target/cov target/debug/$TEST_EXECUTABLE

target/cov/index.html will give a pretty report like this.

Common troubleshooting:

  • If you get 0% coverage in this page, you have an old kcov.
  • If you get a seemingly wrong source code in the detailed coverage, you have run a wrong executable. cargo clean and retry.

I’ve made a simple cargo-cov utility that automates this process for you. This will also remove older test executables. No warranties, use at your own risk.

PKGID="$(cargo pkgid)"
[ -z "$PKGID" ] && exit 1
cargo test --no-run || exit $?
if [ ${#EXE[@]} -ne 1 ]; then
    echo 'Non-unique test file, retrying...' >2
    rm -f ${EXE[@]}
    cargo test --no-run || exit $?
rm -rf $ORIGIN/target/cov
kcov $ORIGIN/target/cov $ORIGIN/target/debug/$PKGNAME-* "$@"


kcov only collects the line coverage. There are many other kinds of code coverage metrics that kcov is unable to gather. Aim for better code coverage, but do not obsess about the exact metric.

There are some issues with the line number, in particular with macros. Also, sometimes, you will get uncovered lines which should not matter (e.g. #[should_panic] tests will get uncovered lines after the panic); kcov does not know about Rust after all.

Code coverage for tests?

Here’s how to tell Travis to do that automatically and upload the result to Coveralls.io.

Put this in your .travis.yml:

language: rust
after_success: |
  sudo apt-get install libcurl4-openssl-dev libelf-dev libdw-dev &&
  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
  tar xzf master.tar.gz && mkdir kcov-master/build && cd kcov-master/build && cmake .. && make &&
  sudo make install && cd ../.. &&
  kcov --coveralls-id=$TRAVIS_JOB_ID --exclude-pattern=/.cargo target/kcov target/debug/<<<MY_PROJECT_NAME>>>-*

Remember to replace <<<MY_PROJECT_NAME>>> with the name of your crate (leave the -* in place, these will match the hash appended to the binary).


That is beautiful! I didn’t even realise that I could just use kcov through Travis CI. :+1:

BTW, @jschievink you have a tiny bug in your .travis.yml. It’s after_success and not after-success.


Whoops! Fixed it.

Should have just copied mine instead :smile:


BTW, is it just me or does code from external crates end up in there for you, too?


Yes, I’ve noticed that too. You can pass filters to kcov that should help with that, but I haven’t tried this yet.


Ah, nice. I had a quick look and adding --exclude-pattern=/.cargo is all that’s needed. Compare this to that.


Great! This also displays source file paths shorter since they now all have a common prefix. I’ve added it to my example .travis.yml above :+1:


For some reason I’m however missing a few source files now. Not sure what’s up with that. They were however missing from the start. So the 95% code coverage isn’t really correct then. :stuck_out_tongue_winking_eye:


Is it possible that code that isn’t touch by any test isn’t even included in the test executable? That could be a possible reason that some files don’t appear in the code coverage report.


How does one interpret what’s going on on line 60 and 61? Also, how do certain lines end up with 1 / 2?


I think the line 61’s coverage is really for line 60 (that is, Ok(r) technically ends right before the { token). Also, the line 58 has a macro try!, which splits the execution (that is, it will execute self.regex(), then an expanded code for try!, then an assignment) and confuses kcov. This is another caveat from using a generic tool with no access to the language knowledge.


(Caveat: I don’t know rust. I am the kcov author though)

kcov uses DWARF debug information to setup breakpoints. It will iterate through all (file,source line) -> address mappings in the debug information and set breakpoints there.

In some cases, there are multiple addresses for a particular line. Sometimes (like the case on line 58), this is probably because the compiler generates two basic blocks for this particular line (part of the line is conditional), so here it simply says that one of the branches has been executed, and the other not. Another case is with inlined code, in which case the same source line can map to many addresses for each of the sites it has been inlined in.

Running kcov on optimized code is of course possible (and advisable where it’s applicable), but can sometimes make for confusing results. Of course, looking at the compiler output will also be confusing then :slight_smile:

As for line 61 (again, not knowing rust!), perhaps the compiler has merged the two Ok calls and just passed different parameters to them?


Some real coverage data from hyper: https://coveralls.io/builds/2464873

We are not that bad: 84% coverage, the generated reports look correct.

EDIT: But some files are missing.


I’ve now integrated this idea into my travis-cargo tool (with @SimonKagstrom’s help): http://huonw.github.io/blog/2015/05/travis-on-the-train-part-2/#covering-your-coverage-needs


To compile on Debian Jessie, install

apt-get install libcurl4-openssl-dev libelf-dev libdw-dev cmake gcc binutils-dev libiberty-dev

To collect coverage the --verify argument is necessary.


This doesn’t upload the test coverage from /tests directory of the project
As an example, my project only has test on its /tests directory and I got a coverage:unknown badge


Would it be prudent to file an issue with the travis people to ask them to make kcov part of the standard install for rust projects? It seems unweildy for them to download it on every rust build. Maybe pass them this thread so they see the use case and need.


Nice information. Thank you!


Good idea, however I’ve noticed that kcov just doesn’t work for some reason for me. Perhaps its same issue ivanceras ran into :confused: ?

I had it working and then after a refactor to put my library on stable it kinda stopped working.