Tutorial: How to collect test coverages for Rust project


#21

I submitted an issue here. It might be possible to find the cookbook or environment generation script that they use, and submit a PR, but I couldn’t find the appropriate place within my timebox (which was an admittedly restricted 10 minutes).


#22

How does one achieve the equivalent of gcc’s profile build in rust?

I’m talking about building with, more or less:

-fprofile-generate -lgcov

and then, after running the binary for a while, rebuilding with:

-fprofile-use -lgcov


#23

That requires compiler support, which doesn’t exist at the moment. (I experimented with it using autofdo to try to generate the data that LLVM needs, but it didn’t work well.)


#24

There is no-sudo way to configure travis for counting coverage and sending it to coveralls.
For example a part of clap .travis.yml:

sudo: false
language: rust
rust:
  - nightly
  - beta
  - stable
before_script:
  - |
    pip install 'travis-cargo<0.2' --user &&
    export PATH=$HOME/.local/bin:$PATH
addons:
  apt:
    packages:
    - libcurl4-openssl-dev
    - libelf-dev
    - libdw-dev
after_success:
  - |
    travis-cargo --only stable coveralls --no-sudo

This will faster your build a lot, because it will run on container-based infrastructure.


#25

I combined a few of the above advices in this travis.yml file

  • It does not require sudo, because it will install kcov in ~/.local/bin instead of /usr/local/bin. This way the new container infrastructure of travis can be used, which will be faster. Thanks to Vinatorui for this advice.
  • I included the --verify option, because I was getting abnormal terminations otherwise. kcov gave a hint to use --verify, which worked for me.

If you use this modified yaml, make sure to change the PKGNAME variable accordingly.

sudo: false
language: rust
rust:
  - stable
  - beta
  - nightly
before_script: |
  PKGNAME="your-project-name" # must match with cargo.toml
  LOCAL="~/.local" # install here to avoid `sudo`
  export PATH=$LOCAL/bin:$PATH
addons:
  apt:
    packages:
    - libcurl4-openssl-dev
    - libelf-dev
    - libdw-dev
    - libbfd-dev # required for `--verify`
after_success: |
  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
  tar xzf master.tar.gz && mkdir kcov-master/build && cd kcov-master/build &&
  cmake -DCMAKE_INSTALL_PREFIX:PATH=$LOCAL .. && make &&
  make install && cd ../.. &&

  kcov --verify \
       --coveralls-id=$TRAVIS_JOB_ID \
       --exclude-pattern=/.cargo \
       target/kcov target/debug/$PKGNAME-*

EDIT: Similarly, I also modified the bash script for local runs of kcov https://gist.github.com/colin-kiegel/e3a1fea04cd3ad8ed06d


#26

I use this script: https://github.com/FractalGlobal/ntru-rs/blob/master/.travis.yml

I also was able to add kcov in the apt repositories for Travis (here), but the version in Ubuntu Precise is not compatible :confused: It should be updated.


#27

BTW, travis-cargo coveralls can do all that in one command, now that I released 0.1.11 which includes support for the --verify flag.


#28

Hi, I’ve tried the method described here, but kcov (v30) seems to be unable to find the debugging symbols in my test executable.

$ kcov --version kcov 30 $ kcov --debug 4 target/cov/ target/debug/trivial-2b7adf9f4183aea6 No debug symbols in target/debug/trivial-2b7adf9f4183aea6. [...]

Of course, I get NaN coverage over 0 lines.

Yet, my executable has debugging symbols, and debugging it in gdb is possible:
$ nm target/debug/trivial-2b7adf9f4183aea6 | wc -l 3487

Any ideas? For reference:
$ rustc --version rustc 1.4.0 $ cargo --version cargo 0.5.0 (built 2015-11-03)


#29

Has anyone tried this for https://codecov.io also?


#30

They don’t support Rust officially. One adapt the C example – if only Rust would generate gcov data. There is a whishlist item for gcov data (#646), but no progress so far.


#31

Alright. I stick with coveralls for now but would be nice with gcov data at some point.


#32

By the way: since Rust supports the MSVC toolchain we should be able to use something like OpenCppCoverage to collect test coverages on Windows. Did anyone try that already? Also, did anyone try getting code coverage using DrCov from DynamoRIO?


#33

What’s the significance of

http://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization

for seeing something similar in Rust?


#34

Hi,

So I use cargo-travis with this config

And I also submitted some testing result here

This tells me I have 94% test coverage which is nice but isn’t really correct because there is a bunch of code in there that isn’t being tested (yet) so I if anyone knows what the issue could be?

Edit:

Now when calling some new code it seems the coverage went down (as not all code is being called)

It seems for some reason that if no code at all in a function is called it’s not included in the stats.

Cheers!


#35

See https://internals.rust-lang.org/t/disabling-gc-sections-when-test-is-specified/2163, with an PR #31368 an important step toward fixing it.


#36

Thanks!


#37

Did anyone have any success with doc test coverage?


#38

Has anyone tried this for codecov also?

Yep, it works! codecov accept the cobertura coverage files, and kcov know how to generate one. I created an issue in travis-cargo to implement that; but for now the travis script is pretty simple.


#39

In case anyone is getting NaN% coverage when trying to run the coverage themselves and having an error in the command line like

kcov-solib: Can't open target/cov//…/kcov-solib.pipe

Please make sure the file system that kcov is writing reports to (target/cov) supports making a pipe (mkfifo). In particular, a shared folder from host in VirtualBox does not support mkfifo. Write to the guest’s storage instead.

(I believe it is a bug in kcov.)


#40

Hi, I’ve tried a git version kcov with my tiny Rust project. It reports the test has 91.2% coverage, but when I click the filename to see what lines aren’t covered, it only shows ??? in various header fields and no code lines. Why is this? I’ve followed this tutorial closely.

Also, run with optimized test results NaN% coverage. Do you get this too?