Integration tests and coverage checks for CLI application

help
1

I've run into something unexpected and I'd love to find the best resolution - at least conceptually. I have a CLI application (here), which has some unit tests and some integration tests. The integration tests are run using assert_cmd, which runs the binary with different parameters, and checks for the desired outcome. This has been working quite well.

I then wanted to check test coverage, and went with the approach recommended in Instrumentation-based Code Coverage - The rustc book. After some tweaking, this also worked reasonably well, but only picked up unit test coverage. Parts of the codebase that were only run as part of integration tests were shown as having no test coverage.

I figured that must be due to the fact that assert_cmd simply runs the CLI binary, so these executions don't really get picked up as executions of functions/regions for profiling purposes. After playing around with the execution environment and various options, I did get some of the integration tests to also show up as coverage, though not in a totally sensible way.

Before I get in too deep (and e.g. go through my actual working/results), I wanted to pose a couple of conceptual questions:

  1. Is my approach to integration testing sensible? What I'm doing feels like the most true E2E integration test... But perhaps a better approach would be to extract most of the functionality into a library (with a public API) and test against that, rather than the CLI? The Integration Tests for Binary Crates section of the book points to this approach. I didn't originally go down this path since the extra layer of abstraction felt unnecessary, given the ability to do E2E tests, but maybe I should reconsider that choice?
  2. Is the "Instrumentation based coverage" approach I linked to what I should be using here, or should I look at alternatives? I'm fairly new to the Rust world, and online searches have turned up various potential approaches, neither of which felt like a slam dunk in terms of usability. But maybe there are other tools out there which more natively handle this workflow?

Thanks for reading. Any input appreciated!

1 Like
2

I chose that route. But I also keep a few E2E tests to actually test that "small" main function.

3

If you're planning on uploading your coverage reports to Codecov or Coveralls or the like, I personally recommend using cargo-llvm-cov for running tests with coverage (e.g., via a GitHub Actions workflow like this). It handles coverage of binary crates run through integration tests, and it's the only Rust coverage tool I've found that leads to actually usable data on Codecov/Coveralls. The only downside is that locally-produced displays of covered code tend to be a bit inscrutable, but I haven't done much with that.

4

Thanks @erelde I'm inclined to think this may be the way to go. Having thought about it a bit more, in my case it would probably also help improve modularity and have some other long-term benefits - so might be worth the change.

@jwodder thanks for the pointer! In case it's not obvious, I'm just getting started with coverage instrumentation, so it's useful to get an idea of the options out there/what others are using. I figured (naively?) that getting something working locally would be the most obvious first step, before potentially shifting over to cloud-based/CI-driven tooling. Though it does seem like the latter may offer better usability out of the box; will check out your suggestions.

5

By the way, tried out cargo-llvm-cov and realized that it's basically a nicely-packaged version of the more DYI-based approach in the Rust book - which is nice.

Still getting some funny results though, like 0% coverage on a module that definitely gets run by the integration tests...will need to dig a little deeper (and maybe refactor as suggested).

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.