Test Coverage with grcov


#1

grcov (https://github.com/mozilla/grcov) is one of the newest Test Coverage tools for Rust created by Mozilla team to gather code coverage results on Firefox.

After a short research I came to conclusion it is the best available tool, even though the cov / cargo-cov (https://github.com/kennytm/cov) looks OK too.

The problem is that there’s very minimal (almost no) usage documentation for it. I would like to use it to gather standard Rust lib.rs project test coverage and send it to CodeCov (https://codecov.io/). The only information provided on this topic is:

Coveralls/Codecov output

grcov ~/Documents/FD/mozilla-central/build -t coveralls -s ~/Documents/FD/mozilla-central --token YOUR_COVERALLS_TOKEN > coveralls.json

and I don’t know what the directories should I used in my case instead of the

  • ~/Documents/FD/mozilla-central/build
  • ~/Documents/FD/mozilla-central should

It would be good to improve the documentation of the grcov tool. I’ve already created a new issue in its issue queue: https://github.com/mozilla/grcov/issues/249


#2

I gave it some time, but with no result so far.

I even tried the cargo-cov (https://github.com/kennytm/cov), but after installing and running it (with clang being installed on my Ubuntu 18.04), the report was empty. Additionally a test with an attribute #[should_panic] failed even though it works with standard cargo test. I commented it out and run withtout it, but the report was still empty.

While gethering some info, I noticed that there is an article “Overview of the Code Coverage Architecture at Mozilla” https://marco-c.github.io/2017/07/28/code-coverage-architecture.html

Please share your successful test coverage experiences with standard/simple Rust library (including instructions would be awsome).


#3

I looked through some crates on crates.io and quite ofthen those having the integration with CodeCov working are using Cargo Travis (https://github.com/roblabla/cargo-travis) for the job.


#4

Did you try cargo-tarpaulin?


#5

Hi Bruno,

Thanks for the tip.

I’ve tried the cargo-tarpaulin and it works fine for me. I’ve got the Test Coverage report and managed to send it from Travis CI (xor-distance-exercise on TravisCi) to CodeCov (xor-distance-exercise on CodeCov), so far the cleanest and simplest solution I’ve tried.

It has a few drawbacks:

  • It requires to run Rust nightly (even though it’s worth a try with Rust stable, not recomended by the authors though).
  • The authors say: “it is still in the early development stage and therefore may contain some bugs”

I’m still interested in finding out how to work with grcov.

I like the cargo-tarpaulin and will use it till (and maybe even after) I find how to make the grcov working.


#6

I’ve just used their example with travis ci and it worked (I think is recent that they put the example).


#7

Hi Mateus,

Thank you for heads up :+1: You are right the “grcov with Travis” example (https://github.com/mozilla/grcov#grcov-with-travis) is quite new, commited just 6d ago (https://github.com/mozilla/grcov/commit/7b3c6e72f1336f788b1f40e91c58ea8dffa318fe).

I’ll try it soon :slight_smile:


#8

So I’ve tried to setup my project based on “grcov with Travis” example (https://github.com/mozilla/grcov#grcov-with-travis), but I had to do couple of changes to RUSTFLAGS.:

  • Change the flag Coverflow-checks=off to Coverflow-checks=on otherwise one of my my tests using #[should_panic] on integer overflow was failing.
  • Remove the flag -Cinline-threshold=0 as all my #[inline] functions were marked as not covered (removing the flag fixed it).

But I still have much worse coverage than before (getting down to 75.76% from 92.11), probably because of:

Check out my CodeCov report: https://codecov.io/gh/dalibor-matura/xor-distance-exercise/tree/87fadb44ffe4d932c383a39c7553040b12654113/src

Related dicsussions:

grcov uses profiling feature -Z profile, built on the gcov-style support in LLVM and obviously there’s still a number of problems with it (not fully baked yet is my assumption based on observation).


#9

I’ve not tried grcov, but if you’re turning on overflow checks, then it seems to me that every overflow check that is not triggered will give you a path that isn’t covered.


#10

Hi Jim,

Thanks for the tip with overflow checks :slightly_smiling_face:


#11

Hey @dalibor I’m also having problems with some “partially covered” that I quite don’t understand. Maybe it will make more sense for you. Could you take a look and see what I’m missing? Or maybe as you mention is just the way that it works for now.

But per example, when trying to access an element in a vector using [index] and also structs that for sure I use in the code. One more per example: to_artists(artists).into_iter().for_each(|artist| {

Here are the links also. I would like to understand how it works because most of my unit tests is covering these things.



#12

Hi @mrcosta,

I’ve got the same problems and wasn’t able to solve them, thus I’m using the cargo-tarpaulin for now.

Quite probably as mentioned by @jimuazu: “every overflow check that is not triggered will give you a path that isn’t covered”, so the question is are you using RUSTFLAGS with Coverflow-checks=off or Coverflow-checks=on?

And as you mentioned it is quite likely that the functionality of -Z profile being used by grcov is not yet fully ready. Here’s the tracking issue for that -Z profile https://github.com/rust-lang/rust/issues/42524 .

I will have to test it all again some day to see what can be done or if it is fixed already.

Cheers,
Dalibor


#13

I’m using Coverflow-checks=off. I will try this other crate and see how it behaves then. I quite don’t get what is -Z profile but I will take a look to understand it. Thanks


#14

Hi @mrcosta,

I’ve already tried to explain it before, but I’m new to the -Z profile too, so I have just some idea about what it is.

The thing is that grcov crate uses profiling feature -Z profile of LLVM

Quoting from https://github.com/rust-lang/rust/pull/42433/commits/42754ce710a777b38402b12728daf54d68ea2b50

Add profiling support, through the rustc -Z profile flag.

When -Z profile is passed, the GCDAProfiling LLVM pass is added to the pipeline, which uses debug information to instrument the IR.