I've run into something unexpected and I'd love to find the best resolution - at least conceptually. I have a CLI application (here), which has some unit tests and some integration tests. The integration tests are run using assert_cmd, which runs the binary with different parameters, and checks for the desired outcome. This has been working quite well.
I then wanted to check test coverage, and went with the approach recommended in Instrumentation-based Code Coverage - The rustc book. After some tweaking, this also worked reasonably well, but only picked up unit test coverage. Parts of the codebase that were only run as part of integration tests were shown as having no test coverage.
I figured that must be due to the fact that
assert_cmd simply runs the CLI binary, so these executions don't really get picked up as executions of functions/regions for profiling purposes. After playing around with the execution environment and various options, I did get some of the integration tests to also show up as coverage, though not in a totally sensible way.
Before I get in too deep (and e.g. go through my actual working/results), I wanted to pose a couple of conceptual questions:
- Is my approach to integration testing sensible? What I'm doing feels like the most true E2E integration test... But perhaps a better approach would be to extract most of the functionality into a library (with a public API) and test against that, rather than the CLI? The Integration Tests for Binary Crates section of the book points to this approach. I didn't originally go down this path since the extra layer of abstraction felt unnecessary, given the ability to do E2E tests, but maybe I should reconsider that choice?
- Is the "Instrumentation based coverage" approach I linked to what I should be using here, or should I look at alternatives? I'm fairly new to the Rust world, and online searches have turned up various potential approaches, neither of which felt like a slam dunk in terms of usability. But maybe there are other tools out there which more natively handle this workflow?
Thanks for reading. Any input appreciated!