Testing crates and best practices


#1

I’m currently investigating what tools are out there that go beyond the built in testing support in Rust. It’s great that we have built in support but coming from Ruby it feels rather rudimentary.

I’ve come across the following crates and I’d be interested to hear everyone’s opinion on them and of course I’m welcoming pointers to tools that I’ve missed:

In addition to that I wonder if there are any “best practices” for writing tests. For example in Ruby I often use dependency injection to replace dependencies of the function under test with more light weight test doubles. Is that something that is done in Rust, too? Maybe by using generics (possibly in combination with trait bounds).

Thanks for your input!


#2

One thing that might be missing is american fuzzy lop in Rust.

Aside from that, there isn’t much in way of testing infrastructure yet (it’s mostly on nightly :frowning: ). The team is still figuring out how to do testing in Rust and it’s non-trivial. Keep in mind Rust wants tests to work for embedded systems, which imposes different set of constraints, than Ruby.


#3

Very cool, thank you! :+1: So AFL is the Rust/compiled languages equivalent to mutation testing?


#4

It’s american fuzzy lop, like the animal :).


#5

AFL is binding in Rust for American fuzzy lop. It’s mostly written in C/C++.


#6

No, AFL fuzzes by generating input and seeing if the program crashes (which it shouldn’t).


#7

Hm, I’m still not entirely sure how this would be used. If you wanted to use it to test a particular struct you would need to create a special executable to convert the data from stdin to whatever you need as input?


#8

That’s how I understand it. AFL is used for testing whole programs.

If you want finer grained testing of the same flavor (i.e., property based testing), then use quickcheck. It operates similarly to AFL. It randomly generates inputs to your property (which is just a function), runs the property, and if it fails, it will take those inputs and shrink them until the inputs cannot be shrunk any more. The output is a witness (the inputs) of failure.


#9

AFL is a little more guided than quickcheck; it actually follows what parts of the binaries are executed (i.e. which branches are taken), and makes efforts to execute the branches that haven’t been taken.

(One could theoretically use it for property testing by setting up individual binaries for each function to test, but this seems like a lot of effort to do manually, in contrast to quickcheck. I imagine a particularly fancy harness could manage setting up and running everything, which would be pretty cool.)


#10

Thanks! Like I said, I’m coming from Ruby so tools like quickcheck or AFL are totally new to me. I will have to give them a try!


#11

Speaking of experiences, what do you miss most (in regards to testing)?

Cucumber? RSpec? Mocks?


#12

What I miss the most is the flexibility. :grin: In Ruby I can pass in any object that just defines the correct methods and it works fine. In Rust I obviously need to use generics or even traits to inject my test dependencies which is of course a bit more work.

I’m currently trying out stainless in combination with hamcrest and that is already an improvement over the built in testing support although using a compiler plugin like stainless feels a bit dirty. Not sure if it should or not though.


#13

Since currently benchmark tests require nightly Rust, then I see no reason to feel dirty.


#14

Only benchmark tests.


#15

Was this a recent change? I could have sworn all tests required nightly.


#16

Not recent, #[test] has been stable & usable since Rust 1.0 (and long before).


#17

Well, tools like stainless introduce completely new syntax as they’re compiler plugins. I haven’t decided yet if that’s something I like or not.