Running coverage for rust lib/app with cargo-make (and in travis too)


#1

Running coverage with rust is not a simple task.
Basically in linux you have to install kcov, run the tests with it using some crazy pattern based for loop and for specific hashes (which changes in every build), and only include your sources in the kcov coverage checks.
At the end, you might also want to upload the results to codecov for example.

With cargo-make this is now a single line command.

Install kcov

For debian based distros you can install kcov with a single command:

cargo make kcov-install-debian

Running Coverage:

cargo make coverage-flow

This basically runs kcov for you against all your unit and integration tests and generate a report under target/coverage.

Travis - Installing, Running and Uploading coverage report to codecov

cargo make ci-flow

This will run the following:

  • cargo build
  • cargo test
  • install kcov
  • run coverage
  • upload to codecov

The last 3 parts for the coverage are only enabled if you define the CARGO_MAKE_RUN_CODECOV=“true” env var.

A simple .travis.yml

language: rust
sudo: true
dist: trusty
group: edge
rust:
  - stable
  - beta
  - nightly
env:
  global:
    - RUST_BACKTRACE=1
    - RUSTFLAGS="-C link-dead-code"
    - CARGO_MAKE_RUN_CODECOV="true"
script:
  - cargo install --debug cargo-make
  - cargo make ci-flow

Could this be any simpler?


#2

Does this work on multi-crate workspaces?


#3

Never tested it but I do have workspace support. Basically what it will do is run same flow for every member including the kcov installation.
So for this to properly work, I need to ensure to prevent kcov from being reinstalled over and over.
The coverage report will be on the member crate level and not merged for all crates together.


#4

Cool. I’m interested because we’ve had a lot of problems getting full coverage to work for rust-rosetta. See #563 for the latest attempt.


#5

ya, that travis cargo integration was something I tried and it seems to be broken. that is part of the reason I wrote this. I am using it inside cargo-make travis build itself, but that is 1 crate and not a workspace.
I guess it will take a bit more work to make it play nicely in a workspace (Like to prevent the mutli kcov installation thing).
Try it out and lets see if we can make it work good. I’m always happy to see PRs and requests as this is a very new project and I am sure many people got a wish list for something but just need to share


#6

Just publish version 0.3.28 which should support workspace level coverage report merge and upload to codecov.


#7

What about llvm/gcov usage, as describe here? Look like it is faster by several orders of magnitude.


#8

I will read about it and see how to integrate it as an optional alternative to kcov.
Shouldn’t be a problem and would just require to setup the tasks for it in the default toml file.
As a task runner cargo make will probably not require any actual code changes.


#9

Added to the github project todos to investigate and implement


#10

Just published a new version (0.3.36) with initial untested support for the llvm cov.
You have to setup the llvm version first for this to be enabled.
I had few issues with this, so I am still checking this (got old version install which might be the issue), but you can look at the full task definition below.
To run this just type

cargo make coverage-lcov

Also added support for tarpaulin coverage tool, to run it:

cargo make coverage-tarpaulin

See more details at: https://github.com/sagiegurari/cargo-make#usage-predefined-flows-coverage

Task Definition (you can modify it in your external Makefile.toml) for the llvm cov :


install_script = [
'''
command -v lcov >/dev/null 2>&1 || {
    if [ "$(grep -Ei 'debian|buntu|mint' /etc/*release)" ]; then
        sudo apt-get update || true
        sudo apt-get install -y clang llvm-3.9 lcov
        
        git clone https://github.com/linux-test-project/lcov.git
        cd lcov
        sudo make install
        cd ..
        rm -Rf ./lcov
    fi
}
'''
]
script = [
'''
#based on https://users.rust-lang.org/t/howto-generating-a-branch-coverage-report/8524
if [ -n "$LLVM_VERSION" ]; then
    rm -f *.gcda *.gcno

    cargo rustc -- --test -Ccodegen-units=1 -Clink-dead-code -Cpasses=insert-gcov-profiling -Zno-landing-pads "-L/usr/lib/llvm-$LLVM_VERSION/lib/clang/$LLVM_VERSION/lib/linux/" "-lclang_rt.profile-${CARGO_MAKE_RUST_TARGET_ARCH}"

    ls *.gcno

    echo "Running unit tests"
    for file in target/debug/deps/${CARGO_MAKE_CRATE_FS_NAME}*
    do
        "$file" || true
    done

    echo "Running integration tests"
    for file in target/debug/deps/test_*
    do
        "$file" || true
    done

    ls *.gcda

    echo "#!/bin/sh -e\nllvm-cov gcov $*" >> ./target/llvm-gcov
    chmod 777 ./target/llvm-gcov
    PATH=$PATH:./target:"/usr/lib/llvm-$LLVM_VERSION/bin"

    LCOVOPTS="--gcov-tool llvm-gcov --rc lcov_branch_coverage=1"
    LCOVOPTS="${LCOVOPTS} --rc lcov_excl_line=assert"
    lcov ${LCOVOPTS} --capture --directory . --base-directory . -o coverage.lcov
    lcov ${LCOVOPTS} --extract coverage.info "$(pwd)/*" -o "${CARGO_MAKE_CRATE_FS_NAME}.lcov"

    genhtml --branch-coverage --demangle-cpp --legend "${CARGO_MAKE_CRATE_FS_NAME}.info" -o target/coverage/
fi
'''
]