Hi, as you know we can write test infinitely. My question is simple: When should we stop? As we all know line coverage is not giving any guarantee so I'm curious about what you guys follow as a metric.
I don't have a metric. All I know is that I write a lot less tests (and a lot less bugs too, for that matter) in Rust than I do in other languages with less expressive type systems and less strict compilers. As to when I stop writing tests, I usually stop when I can't think of any more invariants that need to be upheld by my program or library to test for. When I think of more still untested invariants[1], I add more tests.
Usually only after delivery when I'm in a more relaxed state of mind or when me or someone else discovers a bug (an invariant not upheld for certain state). ↩︎
This is a good question.
One approach I like answers it by inverting the question: "Which test can I write to add the incremental behavior change I need?"
The implementation is only created as the result of a test verifying that behavior:
^^ The book is steeped in "Python, using this specific framework", but the high level ideas seem to apply well to any programming language.
TDD can be very useful, especially if you have the problem “I keep forgetting to write tests”. However, whether it works well depends very much on the character of the program and how finished it is. TDD on a new program in a statically-typed language can go very poorly because a single test implies a large number of new definitions to write out, which can have under-tested implications.
TDD also does not really help with making sure you test edge cases.
All that said, I find it a useful approach for solving the problem “I keep forgetting to write tests when addinf a feature”.
One mechanical way to decide “do I have enough tests” is mutation testing, which can be done in Rust with cargo-mutants. This works by systematically modifying your program to introduce bugs, and then checking whether the test suite fails (detecting the bug); you are done if every such modification is caught. However, this still has false positives (the mutation tool cannot identify when things are modifications that are not bugs) and false negatives (the mutation tool cannot always find a valid way to modify every part of a program).
Never. If you remember the simple quote: every application has at least one bug. Otherwise just follow your intuition: we didn't discover a new bug in the area for long. It means you can stop for awhile.
I like this take on it: Test smarter, not harder - lukeplant.me.uk
In my experience, TDD is really good for when you're fixing bugs, because if you don't have a failing test it's easy to write one after the fact that tests the wrong thing.
But for new stuff it's far more of an art for what's worth it.
personally for nonthrivial code i try to have a test for each case in which the code has a different behaviour like for a function that extract first and last element of a list i test for 0 1 2 and 3 elements.
my teacher recommended to one test for each group of invalid input one for each grroup of valid input and one for each transition point.
ultimately focus on what you think can catch bugs and try to make so you save more time from catching them than it takes you to write and maintain the tests
Most of the answers seem to be focussing on "creating tests for the purpose of testing" I prefer "creating tests to help with thinking, design & high-quality, simple code". (If you're in an area with requirements for MC/DC then that's an extra loop on the bottom of this ...)
I've found that TDD is extraordinarily useful for new stuff in rust - and as with so many things easier to do right in rust than most other languages. It has taken me a while to develop the practical approach to a point where I can reflect on it - this post is my first attempt to sketch out what may well become a blog post or video in the future ...
(This turned out longer than I was expecting, but I found it useful to write. Hopefully others find it useful to read, critique and maybe even use in part)
Anything which is rust-specific (even partly so) is marked [RUST]
I've used rust terminology throughout, as it's defined and consistent within rust - see footnotes for summary definitions
Red - write a failing test for a new API
[RUST]I find I write far fewer unit tests[1] in rust, and I keep even fewer of them mid-term[RUST]Test could be unit test[1:1], integration test[2], example[3]. Personally I don't use doctests[4] for this - for reasons that become clear later- The main goal is to map out the API (either: public - integration test / example, or private - unit test) that I think I want
- Secondarily - test validates the happy path with a concrete, minimal input & concrete output
- Test won't compile as called API does not exist
Green - make the test pass in the easiest possible way
- create the empty types/functions/traits/...
[RUST]add some "note to self" doc comments if there is anything relevant that I want to remember for more than 5 minutes about what this is meant to do, not do, what a pretty version may look like later. Use a# TODOsection with a list of issue numbers & titles for stuff that is not going to be in my focus for the current branch[5][RUST]set visibility as low as possible: only make itpuborpub(crate/super)if this is needed for the test[RUST]forstruct&enum:- only add the fields / variants needed for the test
- only add the impl fn needed for the test
[RUST]fortraitandfn- focus on parameter names, types & return type
- --One of most important rust advantages -- (and the hardest part psychologically)
- if all of the above took more than 30 seconds: the entire impl is
todo!("a 2-3 word, meaningful and unique note")[6] - if there is a chunk of logic identifying different paths based on input: map that out and use
todo!("a 2-3 word, meaningful and unique note")for each path - otherwise hack out a fast, ugly impl. Adding NOTHING which is not needed - e.g. most match arms are
todo!("a 2-3 word, meaningful and unique note"), don't parse anything which doesn't need it yet, .. - add
#[should_panic(expected = "a 2-3 word, meaningful and unique note")]
- if all of the above took more than 30 seconds: the entire impl is
- run the test. It passes (assuming I got the branching logic correct and the API doesn't need to change based on what I noticed while implementing)
- commit
Refactor
[RUST]Optionally repeat Red/Green if using#[should_panic]- change / remove the
#[should_panic] - finish (some of) the logic
- change / remove the
- Adjust the code style, API - the main aim is to reduce the total lines of code until it feels OK considering it's only partly done
- Test & commit regularly in a loop
[RUST] Red & Green - write a new should_panic test
- Pick the next most important
todo! - Add a test specifically designed to trigger that case; if this feels convoluted - pick another
todo!, this one may disappear on its own soon - Add anything needed to make this compile but no new logic
- test, passes, commit
Refactor
- Make the change easy, then make the easy change
[RUST]pull out some parsing logic to a type &TryFrom[RUST]move a bare fn to a direct method or direct method to a trait- etc ...
- if this breaks a unit test[1:2] because it changes an internal API.
- Ideally add an integration test, if there isn't one already, if it passes: delete the offending unit test otherwise mark both as
#[should_panic(expected=...)] - If you're not there yet and focussing on internal API - stash the change; update the test to what you think the new internal API should be; run test & check it fails correctly; pop the stash; test & commit when passing
- Ideally add an integration test, if there isn't one already, if it passes: delete the offending unit test otherwise mark both as
[RUST]Optionally repeat Red/Green if using#[should_panic]- change / remove the
#[should_panic] - finish (some of) the logic
- re-run the test, it passes
- commit
- change / remove the
- Adjust the code style, API - the main aim is to reduce the total lines of code until it feels OK considering it's only partly done
- Test & commit regularly in a loop
Repeat until you have no more todo!(...)
- Some will have disappeared as you refactored to a simpler style (YAGNI beats DRY - you saved all the effort of coding, testing & maintaining paths which aren't needed)
- Feel free to delete tests which helped earlier and now just restrict your ability to move as long as you already have another test which executes the same thing in a better way. I often find I start the cycle with 2-3 unit tests, then delete at least one of them when I'm refactoring based on integration tests
- Your API is nice & clear, minimal, documented well through tests & possible external examples[3:1]
- You have full bi-directional coverage: every test tests something meaningful and will fail only if needed; all your code is tested - any changes which break the code will make a test fail
Refactor before release
[RUST]change the doc comments from "note to self" to "a clear explanation of intent"[RUST]add doctests[4:1] for (almost) everything- this forces you to go through and read the entire relevant code as a unit, not a series of small changes
- pull the code into a shape that you like - you may do some serious extraction & deletion in this phase. Think of it like sanding the edges on a piece of wood which just before attaching it or declaring the whole thing "done"
- add any extra tests for edge cases you notice which "should already work but..."
[RUST]trust the compiler & clippy - so many things will fail to compile or oncargo clippy -- -D warningsthat you don't need to construct extra tests for them
At this point you have the answer to
"Now"
test is part of the crate - i.e.
mod tests { use super::*; ... }↩︎ ↩︎ ↩︎test is in a separate
testsdirectory - compiled as an external crate i.e. requiresuse my_crate::{Struct1; Trait1; ...}↩︎in separate
examplesdirectory, designed as a showcase how-to/cookbook/etc. & compiled as an external crate i.e. requiresuse my_crate::{Struct1; Trait1; ...}- by default only validates "will this compile" unless extra#[test]s are included andcargo test --examplesis run specifically ↩︎ ↩︎code examples inside a
///or//!block - compiled as an external crate but directly inline in the main src file, shown on docs.rs ↩︎ ↩︎I use the vscode github plugin which lets me create an issue & adds the number to the TODO comment ↩︎
no two notes are the same, worst case I throw in the current line number ↩︎
As you noted, it's impossible to prove the correctness of software by testing. I conclude:
a) One stops testing when one runs out of time, money, effort, will power.
b) Every run of your program in production forever after is a test.
I worked for some years testing aircraft flight control software, think Boeing 777 Primary Flight Computers, C-130 Hercules actuator control, Rolls-Royce engine management systems, etc. We had to achieve 100 code coverage in those tests. But more than that we had to test every decision the program made (in conditional statements, loops etc) went the right way for given the data it was evaluating. For example does a range check of a value actually jump the right way at the right value or is there an off by one error? This gets to be a lot of tests when a condition has multiple expressions to evaluate.
Then there is the issue of timing. Typically in a control system one has a strict time limit on how much time each iteration of a control loop can take, say 10ms.
As you can image this involved whole teams of people testing for many man years.
Regarding b) above. If every run of your program in production is a test one might want to think about what happens when it fails. In the simple case of a panic is one going to arrange to restart the program? How is one going to report the problem so that it can be investigated? Etc.
Worse is when the code runs but produces incorrect results. Is one going to have some other system to check against?
Nowadays the tings I work on are not so critical. I'm writing it in Rust. So if it compiles it's good to go 
It is true that it is possible to write many tests checking trivial things repeatedly without much use, but this does not happen if you pay attention to the test coverage.
cargo install cargo-llvm-cov
cargo llvm-cov --html
Checking test coverage allows to understand where the real tests are missing, and which code is already covered well enough. This tool may also discover lines of your code that you absolutely cannot test because that code .. is dead, never runs.
Generally the more test coverage, the better, even if some say that values much above 90 % may have diminishing returns. Enough tests allow to refactor the code safer, as the testing suite protects against regressions.
Many developers write small evaluation fragments to check if the new code under development works. These can be later discarded, or can be converted to test instead. Same about bug reports from programming users - not all are finished tests, but can usually be completed into tests.
A can agree with that.
To my mind aiming for 100% or some high code coverage is useless, even misleading, if those tests don't have any other reason to exist than achieving coverage.
A simple example:
if input > SOME_LIMIT {
do_stuff();
} else {
do_other_stuff();
}
It easy to get 100% coverage here. Just:
a) Have a test with input at some high value and check stuff happens.
b) Have a test with input at some low value and check other stuff happens.
But that leaves so much untested. Is SOME_LIMIT the correct value? Is that really supposed to be a greater than comparison, a greater or equal comparison or something else?
Its gets worse rapidly if one has complicated expressions involving many values in the condition and logical operators. For example:
let account_age_months = 14;
let total_purchases = 150.50;
let monthly_active_days = 22;
let support_tickets_open = 0;
let has_banned_history = false;
if (!has_banned_history && support_tickets_open <= 1)
&& (account_age_months >= 12 || total_purchases > 500.0)
&& (monthly_active_days * 4 >= 60)
{
println!("Upgrade eligible: User meets all multi-factor security and activity metrics.");
} else {
println!("Upgrade denied: Criteria not met.");
}
´´´
It's still the same easy to get 100% coverage but have you really verified that expression works correctly?100% coverage for a whole binary is absolutely a waste of time.
One thing I've seen work decently well, though, is insisting on 100% coverage of certain crates. You just make sure that the "glue" is in a different crate, and thus the non-glue library crates should be 100% hit-able without too much trouble from tests (with a couple exceptions for things like _ => unreachable!() arms that truly are unreachable).
This is what branch or path coverage is for.
scnr
Yes indeed. But what if you only have one path and no branches. My example above could be written branchlessly:
let account_age_months = 14;
let total_purchases = 150.50;
let monthly_active_days = 22;
let support_tickets_open = 0;
let has_banned_history = false;
let is_eligible: bool = (!has_banned_history && support_tickets_open <= 1)
&& (account_age_months >= 12 || total_purchases > 500.0)
&& (monthly_active_days * 4 >= 60);
let index = is_eligible as usize;
let outcomes = ["Upgrade denied: Criteria not met.", "Upgrade eligible: User meets all multi-factor security and activity metrics."];
println!("{}", outcomes[index]);
This only needs one test to run the code once to achieve 100% coverage. Which by itself is totally useless.
In the definition I know, that's 5 paths needing coverage.
Sure. The problem @ZiCog refers to is that coverage tools, such as the one mentioned above that we have for rust, will rate code as in @ZiCog's most recent example 100% covered with one appropriately formulated test. As your point implies, separate tests can and maybe should be written for all the permutations of the compound condition in the example.
One option is to rewrite the code so that every logical permutation is a code branch. Then the code coverage tool won't give you 100% till you test all the component conditions. That can make the code easier to test, even when a code coverage tool is not used.
Personally, I don't think we should be overly fixated on high test coverage numbers.
Tests are valuable for catching compile-time errors and certain logical issues, but they can't cover everything — especially runtime behavior, concurrency bugs, or edge cases in real-world usage. In my experience, chasing 90%+ coverage often leads to writing tests that add little value while slowing down development.
I used to obsess over coverage metrics too, and it ended up delaying projects more than once. Over time, I've learned that test quality matters much more than quantity. Focusing on key scenarios, boundary conditions, and integration points tends to give better returns than trying to cover every line.
Hope this is useful to you.
The talk about "how do you know when you're done writing tests" reminded me of a talking point about the 2015 game Her Story - which is basically a bunch of short video clips with links that you navigate between that together, as the name implies, tell a story. While I can't find any canonical source the distilled version is a forum conversation about it went something like:
> How do I know when I've finished the game?
> When you've seen enough that you feel satisfied.
> How do I know when I'm satisfied?
Funny, but it does actually speak to a real issue - it's hard to feel satisfied that you've "gotten everything" in such a game without a checklist. You kind of just have to accept that what's important to know is entirely up to you.
Back in the testing world we also don't have a reliable version of a checklist either (coverage as discussed above is a fairly flawed proxy) but you can decide what matters for your context and design tests that check that. Does it matter if a network request could block the application startup for several seconds if the user has some firewall misconfigured? Does it matter if the disk is touched even when configuration hasn't changed? Does it matter if it crashes with memory exhaustion when fed a 10TB input file?
For most programs "correctly handling" "every" situation is neither necessary nor plausible, but most programs will have at least some edge cases like the above that don't matter for most cases. The best you can do is develop experience about what sorts of things are likely to matter and how to test for them.
Just picking this as the latest similar statement.
To my mind if you are going to measure code coverage - the only viable expectation is 100%[1] anything else is just noise.
I treat test coverage the way I do lints - usually valuable, sometimes needs silencing, with an explanation. So I have no problem with code that includes pragma-like markers with explanations - for rust that's the #[coverage(..)] attribute.
If I have coverage enabled - then I have a CI expecting 100% and expect to see sections specifically excluded with reasons. That's the only way I know that the coverage is "right"[2]
FWIW I almost never look at test coverage in rust - so much of what it would tell me is already covered by the compiler and clippy that it's not worth it compared to most other languages. I also have far fewer tests in rust for the same reason - there's a load of stuff that doesn't need testing because it just won't compile / lint.
If by "measure" you mean caring about the percentage, sure: generally I find coverage mostly useful as a guide to what "obvious" tests I'm missing, or as a hint when debugging tests.
Alternatively, you can have a CI check that coverage shouldn't go down as proxy for new features missing tests, but that should only be a hint for reviews, like CI benchmarking.
The general point, though, is that chasing that 100% is neither necessary nor sufficient, and to keep in mind it's only a proxy for how well tested your code is.