Fuzzing with cargo fuzz


#1

Continuing the discussion from What's everyone working on this week (45/2017)?, I did finally find a bug in that crate I was working on, when I fed the fuzzer a custom dictionary (in this case with fibonacci words, so the input was periodic enough to find bugs that depended on that.)

I found a slide deck with an overview of libfuzzer (which is what cargo fuzz helps you use), it’s this pdf. It really has a range of cool features for example automatic dictionaries by intercepting memcmp and strcmp.


#2

That’s pretty cool!

In the past I’ve found fuzzers to be pretty good at finding the low hanging fruit or random edge cases you’d never think to check when writing tests manually. You can usually provide it with some corpus of “valid” inputs that the fuzzer can use as starting points.

I think the best bug I’ve found was a case where the parser for some custom binary file format would recurse infinitely by trampolining between two functions. It was a silly mistake where I forgot to add a proper base case (I wasn’t processing any input, so it’d see a token then go to a rule which would then see the same token and jump back to the previous rule, ad infinatum) but it’s still nice to know there are tools out there to help test scenarios you may not think about.


#3

I also think that this is where fuzzers shine: they can find bugs which neither a white-box nor a black-box tester would have ever thought about. By virtue of randomness, they can think outside the box, so to speak.

Now, they have other undesirable characteristics, such as non-determinism, lack of intelligence in the walk through the input space, poor analysis of the test results (“does not crash or invoke UB” is a very low standard for working code, and if you can define more precisely the behaviour of a working system, you can write the test yourself), or an intrinsically slow and wasteful mode of operation, which is why IMO they should always be used as a last resort option after everything else has been tried.

But for sensitive code which needs to process fully untrusted user input (e.g. bytes from a network), which has more internal complexity than a human can grasp, or which has never been subjected to proper testing when it was initially written, they seem like a good complement to other testing methodologies.


#4

By the way, if anyone is trying out cargo fuzz right now, it’s broken on the latest nightlies. For now it’s “simple” to work with it anyway by having it use the latest working nightly. Info on which version to use is in this issue. For example this one works: nightly-2017-10-04.