Wherein we look at what needs to be done to get mutation testing in Rust.
I did know what fuzzing is, but I didn’t realize how much it is until I’ve seen this talk. It helped me understand that unit testing, smoke testing, and so on isn’t enough
One thing that might be feasible for the early return stuff is to steal a march from QuickCheck, and return an
Arbitrary instance of a type if you can.
…Actually, could one replace some set of (sub-)expressions with
Arbitrary::arbitrary(), wrapped in something that controls it using the mutation counter? Something like
mu::choose(original_code, Arbitrary::arbitrary, mc, 14)
That might actually be fully general - returning an arbitrary type, etc, using the same machinery the user (hopefully) already wrote for property-based testing.
I’m not sure if that really works (or has a sufficient chance of not failing the tests). My current idea is to use a lint to identify one of the following options to get a value of the respective type:
- an argument-less constructor function
- a binding of the same type within the function (including identifying the scope where it is available)
Is the idea here to try to do something similar to “american fuzzy lop”? http://lcamtuf.coredump.cx/afl/
Similar, but not quite alike. The idea is to run the original test suite, then introduce one error at a time and note which tests fail. If an error isn’t caught by the tests, report it, so the user can cover it with another test. Otherwise use minimum set cover to find tests that can be removed (according to some weight function) and report them, too.
Alright. Sound good. I’m looking forward to your progress on this