PSA: code coverage reported by `travis-cargo {coveralls,coverage}` may suddenly drop (become more accurate)

announcements
1

Thanks to some great work by @euclio, I just released travis-cargo 0.1.15.

This release will pass -C link-dead-code to the compiler on nightly and beta (and stable from the 1.9 release) when recording code coverage, which makes the code coverage numbers more accurate. Previously functions that were never called at all were likely to be not counted for code coverage in any way, resulting in under-reported coverage numbers (see the discussion thread on internals for more details). Now, these functions are included, and all count as uncovered.

For example, the following two jobs are the exact same source file, the only difference is the compiler version. The first used a compiler that does not support -C link-dead-code while the second used one does:

  1. huonw/travis-cargo | Job 14 | src/lib.rs | Coveralls - Test Coverage History & Statistics 100% coverage
  2. https://coveralls.io/jobs/14197542/source_files/398912084: 73% + big red marks over the function3, which isn't tested.

(The last few functions in that file aren't covered with either compiler because they are killed by #[cfg]s: they never really existed.)

Hopefully this makes code coverage a more useful measure/tool. Don't be surprised if your numbers drop suddenly!

5 Likes
2

Something I keep meaning to ask, but haven't yet; do doc tests count as coverage? Or only those tests marked with #[test]?

1 Like
3

They're not counted, as the way in which they are executed is pretty complicated: they're not just normal binaries in cargo's mind or on disk, meaning it's not nearly as easy to consider them as other tests.

4

Hi,

I'm running with Beta now and I have one question:

This line gets marked as non-covered and while I know that the actual code is being used i'm a bit curious of the reason why that is?

Cheers!

5

I'm not sure: it may be a bug in the debug-info rustc outputs (so kcov doesn't see execution hit that line), or maybe a bug in kcov. It probably isn't something travis-cargo itself can influence/fix, unfortunately.

6

Yeah I would assume it's something along those lines. Not a problem for me but just a bit curious about it.