Test Coverage with grcov

help
1

grcov (GitHub - mozilla/grcov: Rust tool to collect and aggregate code coverage data for multiple source files) is one of the newest Test Coverage tools for Rust created by Mozilla team to gather code coverage results on Firefox.

After a short research I came to conclusion it is the best available tool, even though the cov / cargo-cov (GitHub - kennytm/cov: LLVM-GCOV Source coverage for Rust) looks OK too.

The problem is that there's very minimal (almost no) usage documentation for it. I would like to use it to gather standard Rust lib.rs project test coverage and send it to CodeCov (https://codecov.io/). The only information provided on this topic is:

Coveralls/Codecov output

grcov ~/Documents/FD/mozilla-central/build -t coveralls -s ~/Documents/FD/mozilla-central --token YOUR_COVERALLS_TOKEN > coveralls.json

and I don't know what the directories should I used in my case instead of the

  • ~/Documents/FD/mozilla-central/build
  • ~/Documents/FD/mozilla-central should

It would be good to improve the documentation of the grcov tool. I've already created a new issue in its issue queue: Documentation improvement Ā· Issue #249 Ā· mozilla/grcov Ā· GitHub

7 Likes
2

I gave it some time, but with no result so far.

I even tried the cargo-cov (GitHub - kennytm/cov: LLVM-GCOV Source coverage for Rust), but after installing and running it (with clang being installed on my Ubuntu 18.04), the report was empty. Additionally a test with an attribute #[should_panic] failed even though it works with standard cargo test. I commented it out and run withtout it, but the report was still empty.

While gethering some info, I noticed that there is an article "Overview of the Code Coverage Architecture at Mozilla" Overview of the Code Coverage Architecture at Mozilla | Marco Castelluccio

Please share your successful test coverage experiences with standard/simple Rust library (including instructions would be awsome).

3

I looked through some crates on crates.io and quite ofthen those having the integration with CodeCov working are using Cargo Travis (https://github.com/roblabla/cargo-travis) for the job.

4

Did you try cargo-tarpaulin?

1 Like
5

Hi Bruno,

Thanks for the tip.

I've tried the cargo-tarpaulin and it works fine for me. I've got the Test Coverage report and managed to send it from Travis CI (xor-distance-exercise on TravisCi) to CodeCov (xor-distance-exercise on CodeCov), so far the cleanest and simplest solution I've tried.

It has a few drawbacks:

  • It requires to run Rust nightly (even though it's worth a try with Rust stable, not recomended by the authors though).
  • The authors say: "it is still in the early development stage and therefore may contain some bugs"

I'm still interested in finding out how to work with grcov.

I like the cargo-tarpaulin and will use it till (and maybe even after) I find how to make the grcov working.

1 Like
6

I've just used their example with travis ci and it worked (I think is recent that they put the example).

1 Like
7

Hi Mateus,

Thank you for heads up :+1: You are right the "grcov with Travis" example (https://github.com/mozilla/grcov#grcov-with-travis) is quite new, commited just 6d ago (https://github.com/mozilla/grcov/commit/7b3c6e72f1336f788b1f40e91c58ea8dffa318fe).

I'll try it soon :slight_smile:

1 Like
8

So I've tried to setup my project based on ā€œgrcov with Travisā€ example (https://github.com/mozilla/grcov#grcov-with-travis), but I had to do couple of changes to RUSTFLAGS.:

  • Change the flag Coverflow-checks=off to Coverflow-checks=on otherwise one of my my tests using #[should_panic] on integer overflow was failing.
  • Remove the flag -Cinline-threshold=0 as all my #[inline] functions were marked as not covered (removing the flag fixed it).

But I still have much worse coverage than before (getting down to 75.76% from 92.11), probably because of:

  • CodeCov complaining about "Partially covered lines" which in number of cases looks good to me. I'm still investigating it.
  • The ending curly bracket } of my #[should_panic]` tests is marked as not covered (e.g Codecov).

Check out my CodeCov report: Code coverage done right.

Related dicsussions:

grcov uses profiling feature -Z profile, built on the gcov-style support in LLVM and obviously there's still a number of problems with it (not fully baked yet is my assumption based on observation).

2 Likes
9

I've not tried grcov, but if you're turning on overflow checks, then it seems to me that every overflow check that is not triggered will give you a path that isn't covered.

1 Like
10

Hi Jim,

Thanks for the tip with overflow checks :slightly_smiling_face:

1 Like
11

Hey @dalibor I'm also having problems with some "partially covered" that I quite don't understand. Maybe it will make more sense for you. Could you take a look and see what I'm missing? Or maybe as you mention is just the way that it works for now.

But per example, when trying to access an element in a vector using [index] and also structs that for sure I use in the code. One more per example: to_artists(artists).into_iter().for_each(|artist| {

Here are the links also. I would like to understand how it works because most of my unit tests is covering these things.

12

Hi @mrcosta,

I've got the same problems and wasn't able to solve them, thus I'm using the cargo-tarpaulin for now.

Quite probably as mentioned by @jimuazu: "every overflow check that is not triggered will give you a path that isnā€™t covered", so the question is are you using RUSTFLAGS with Coverflow-checks=off or Coverflow-checks=on?

And as you mentioned it is quite likely that the functionality of -Z profile being used by grcov is not yet fully ready. Here's the tracking issue for that -Z profile https://github.com/rust-lang/rust/issues/42524 .

I will have to test it all again some day to see what can be done or if it is fixed already.

Cheers,
Dalibor

2 Likes
13

I'm using Coverflow-checks=off. I will try this other crate and see how it behaves then. I quite don't get what is -Z profile but I will take a look to understand it. Thanks

14

Hi @mrcosta,

I've already tried to explain it before, but I'm new to the -Z profile too, so I have just some idea about what it is.

The thing is that grcov crate uses profiling feature -Z profile of LLVM

Quoting from Build instruction profiler runtime as part of compiler-rt by marco-c Ā· Pull Request #42433 Ā· rust-lang/rust Ā· GitHub

Add profiling support, through the rustc -Z profile flag.

When -Z profile is passed, the GCDAProfiling LLVM pass is added to the pipeline, which uses debug information to instrument the IR.

15

Have added a PR to make it clearer how to use grcov out of the box:

Please critique!

2 Likes
16

grcov currently supports incomplete support and unsupported branch coverage. In addition, there are some problems with incorrect test results.
https://github.com/mozilla/grcov/issues/725

17

Please don't bump old threads.

18

This topic was automatically closed after 24 hours. We invite you to open a new topic if you have further questions or comments.