Why does Rust test framework lack fixtures and mocking?

1

As a newcomer to Rust, it surprises me that the Rust test framework is woefully deficient. One of the most important issues when learning a new language for me is how good the test framework is. Now I know there are other 3rd party test frameworks like stainless, but this requires the nightly build of Rust. As a Rust beginner, this is not a practical route to go down, I need a stable platform to work on, without having to trouble shoot issues (which I don't currently have the skills to cope with) that may occur as a result.

A test framework, without fixtures is next to no use at all. And Ideally, a mocking framework also needs to be included; genuine TDD is not practical without these features, and I feel a bit frustrated in having to refer to 3rd party solutions, when learning Rust is difficult enough as it is.

Is there a plan to get these features into the stable release of Rust?

6 Likes
2

Let me just say that "nightly" means a different thing for Rust than in many other contexts. For some of my projects I've used nightly-only for months now, and I think I had exactly one problem with a bug in the compiler itself.

And this is no accident: since you can use newly added (so called "unstable") features only on nightly, it has to be consistently usable for people to evaulate the features' usefulness and ergonomics.

As for getting features into stable: without knowing stainless in detail, I can see that it uses compiler plugins for procedural macros. This API is planned to be stabilized, but there is no schedule for that as far as I know.

1 Like
3

It's early here and I'm not an expert, but: I would not call Rust's/Cargo's testing support a 'framework', because then people tend to compare it to RSpec. There are annotations and conventions for where to put tests (and benchmarks), but that's basically it. In the end, a #[test] just defines a bit of code that will be executed in a thread and it will see whether that thread panics.

What specific use cases are you thinking about? While there is no big testing framework and you may need to write more boilerplate yourself, Rust's tests are definitely usable.

What does 'fixtures' mean to you? Running some setup code and creating some struct instances to test? You can call a setup function in each test (defined in the file itself or you can include a whole source file full of setup functions into the current scope with include!), maybe returning a tuple of items to work with. In unit tests, you can access the whole parent scope, so you can use the private fields/methods of your structs and traits (to e.g. directly create a struct instance by setting all fields).

Mocking is quite possible using traits. A trivial example: You want to read a file's content and parse it. Then instead of fn parse(file: File) -> Magic, you could do fn parse<R: Read>(input: R) -> Magic and accept any kind of reader. This way, you can, in your tests, just pass it another type that implements Read (e.g., an array of bytes).

There are some very well tested Rust projects out there, e.g. Cargo itself (integration tests for a CLI tool as well as the core functionality) and Diesel (tests with database setup and compile tests for the query builder).

2 Likes
4

Hi Killer cup. Thanks for your response. As I said before, I am new to rust, so when I browse, existing Rust repos, the code looks too complicated to get my head round. I want to avoid trying to run before I can walk. As for fixtures, all I want is to be able to define Setup and Teardown functions for each test, and then some external process (the rust test framework, for arguments sake) to call them. Since this does not currently exist, I am not sure how I get this functionality. Your paragrapgh beginning with 'What does 'fixtures' mean to you?' doesn't make much sense to me. It sounds like to me, that you're expecting that I actually code up a test runner. That's exactly what I don't want to do, because that sounds complicated given that I'm still learning rust (ps, I am not a newbie programmer, just a newbie rustie). The facility that cargo test gives me, is that it will invoke all functions with a test attribute, when what I want is a bit more than than so I can get full test fixture functionality. So far, I have not seen that in any of the tests in Cargo test. In fact I'm looking at cargo tests, and I can see a lot of empty setup functions, then a lot of large functions using a test! macro, but nothing that defined a test using the test attribute, that I know about, so its very frsutrating. I guess I'm going to have to learn to run before I can walk, and start reverse engineering cargo tests, since at leat there are a lot of tests to look at, but this seems to me to be putting the cart before the horse. :frowning:

5

What about quickcheck: http://burntsushi.net/rustdoc/quickcheck/index.html ?

6

I'm sorry, @Zephilim, I should've been clearer. Let me try to answer some of your questions:

Sure, that's totally reasonable! Using a complex testing framework is a bit like running, though, right? Especially in Rust, where it will probably involve macros, or even (unstable) compiler plugins. That shouldn't stop you from learning, though!

I haven't seen any crate that automates this, aside from stainless (unstable compiler plugin…). So, you'll have to write a bit more boilerplate.

A test runner? No, cargo test will do that for you (in the threaded way I tried to describe above). I was talking about:

#[test]
fn your_test() {
    setup();

    assert!(1 < 2);

    teardown();
}

fn setup() { /* ... */ }
fn teardown() { /* ... */ }

I mentioned cargo specifically because they include the test! macro that you saw to do setup and teardown stuff. You can find the sources here (I searched for macro_rules! test).

1 Like
7

Can the test attribute be set on member functions on a struct? In the setup and teardown, I would need to init and destroy state, for use in the body of a test. Nothing I've read or seen in any repo suggests this is possible (but I suppose I could try it out, but I can't believe that I am the only person that would need this, afterall, this is the prevailing methodology used in xUnit test frameworks). As I type this, a whole host of issues have cropped up in my mind; ie the setup and teardown would need to be on an instance; so how would you create that instance, arghhh. (Sorry I'm just thinking out loud ..., but I'm sure you get the gist)

8

Many of the fundamental libraries don't require very sophisticated test frameworks, so I'm not surprised that there isn't a multitude of such frameworks yet. But I don't think it will stay that way.

If you want to get an idea how more complex tests could look, maybe have a look at Diesel's tests? It at least has to set up a database schema and connection for many of them.

One thing you might have to adjust is the expectation of pervasive object-oriented style. For example, instead of having tests as a method on a TestSuite type struct you typically have test functions, which create the required state themselves or call a helper function to create the state.

Ah, and don't be afraid to use macros, this is exactly the scaffolding that they are a good match for, in order to reduce boilerplate. A non-nightly test framework written today would probably make extensive use of macros.

9

argh, so that is where the test macro is. I was working my way down the files in the test dir, and its just typical that it the last file in that long list of test files.

10

Ok, I'm going to deep delve into cargo and diesel, and see what else I can do to start writing production quality real world unit tests.

11

Might be worth that Rust itself has quite a bunch of tests so browsing the Rust repro might give some ideas. Here is a bunch of tests for fs for example https://github.com/rust-lang/rust/blob/master/src/libstd/fs.rs#L1460

12

The Rust test crate was designed to be very minimal, but to be extended in the hypothetical future by more advanced test frameworks.

Here's a recent discussion on extensible test frameworks that at least hints at the direction we want to go in, but there's nobody working on it right now.

In the meantime, if you don't want to pull in a crate to add test fixtures, there's a simple pattern for setup and teardown: create a function called setup that does the setup and accepts a closure to run the test, and passes it any common state. Then your test looks like

#[test]
fn make_sure_foo_works() {
    setup(|&fixture_state| { ... });
}
2 Likes
13

Hi brson, thanks for that, very useful. [newbie mode] I just tried your prosposed work around, and can't quite get it right. This is a mockup:

#[cfg(test)]
mod fixt {

    struct TestFixture { name : String }

    fn Setup( o: fn (tf : TestFixture) ) {
         o.name = "Initialised";
     }

    #[test]
    fn make_sure_foo_works() {
        Setup(| &x | {
            assert_eq!(x.name, "Initialised");
        } );
    }
}

results in the following errors:

c:\dev\sandpit\rust\expressions>cargo test
Compiling expressions v0.1.0 (file:///C:/dev/sandpit/rust/expressions)
src\tests\fixt.rs:8:10: 8:16 error: attempted access of field name on type fn(tests::fixt::fixt::TestFixture), but no field with that name was found
src\tests\fixt.rs:8 o.name = "Initialised";
^~~~~~
src\tests\fixt.rs:14:24: 14:30 error: the type of this value must be known in this context
src\tests\fixt.rs:14 assert_eq!(x.name, "Initialised");
^~~~~~
src\tests\fixt.rs:14:13: 14:47 note: in this expansion of assert_eq! (defined in )
:5:8: 5:18 error: the type of this value must be known in this context
:5 if ! ( * left_val == * right_val ) {
^~~~~~~~~~
src\tests\fixt.rs:14:13: 14:47 note: in this expansion of assert_eq! (defined in )
:5:22: 5:33 error: the type of this value must be known in this context
:5 if ! ( * left_val == * right_val ) {
^~~~~~~~~~~
src\tests\fixt.rs:14:13: 14:47 note: in this expansion of assert_eq! (defined in )
src\tests\fixt.rs:13:15: 15:10 error: mismatched types:
expected fn(tests::fixt::fixt::TestFixture),
found [closure@src\tests\fixt.rs:13:15: 15:10]
(expected fn pointer,
found closure) [E0308]
src\tests\fixt.rs:13 Setup(| &x | {
src\tests\fixt.rs:14 assert_eq!(x.name, "Initialised");
src\tests\fixt.rs:15 } );
src\tests\fixt.rs:13:15: 15:10 help: run rustc --explain E0308 to see a detailed explanation
error: aborting due to 5 previous errors
Could not compile expressions.

So, I'm not quite there, can you assist? Thanks.

14

Try making the name field public. So struct TestFixture { pub name: String }. That'll fix the complaint about there not being a field called name.

The closure issue is different though, and I'm on my phone so can't mess with that one...

15

I realised in the Setup, I also need to call into the lambda. I changed name to be public so it now looks like this:

#[cfg(test)]
mod fixt {

    struct TestFixture {
        pub name : String
    }

    fn Setup( o: fn (tf : TestFixture) ) {
         o.name = "Initialised";
         tf(o);
     }

    #[test]
    fn make_sure_foo_works() {
        Setup(| &x | {
            assert_eq!(x.name, "Initialised");
        } );

        // Do a similar thing here with a teardown
    }
}

but now I get:

c:\dev\sandpit\rust\expressions>cargo test
Compiling expressions v0.1.0 (file:///C:/dev/sandpit/rust/expressions)
src\tests\fixt.rs:11:10: 11:12 error: unresolved name tf [E0425]
src\tests\fixt.rs:11 tf(o);
^~
src\tests\fixt.rs:11:10: 11:12 help: run rustc --explain E0425 to see a detailed explanation
src\tests\fixt.rs:10:10: 10:16 error: attempted access of field name on type fn(tests::fixt::fixt::TestFixture), but no field with that name was found
src\tests\fixt.rs:10 o.name = "Initialised";
^~~~~~
src\tests\fixt.rs:17:24: 17:30 error: the type of this value must be known in this context
src\tests\fixt.rs:17 assert_eq!(x.name, "Initialised");
^~~~~~
src\tests\fixt.rs:17:13: 17:47 note: in this expansion of assert_eq! (defined in )
:5:8: 5:18 error: the type of this value must be known in this context
:5 if ! ( * left_val == * right_val ) {
^~~~~~~~~~
src\tests\fixt.rs:17:13: 17:47 note: in this expansion of assert_eq! (defined in )
:5:22: 5:33 error: the type of this value must be known in this context
:5 if ! ( * left_val == * right_val ) {
^~~~~~~~~~~
src\tests\fixt.rs:17:13: 17:47 note: in this expansion of assert_eq! (defined in )
src\tests\fixt.rs:16:15: 18:10 error: mismatched types:
expected fn(tests::fixt::fixt::TestFixture),
found [closure@src\tests\fixt.rs:16:15: 18:10]
(expected fn pointer,
found closure) [E0308]
src\tests\fixt.rs:16 Setup(| &x | {
src\tests\fixt.rs:17 assert_eq!(x.name, "Initialised");
src\tests\fixt.rs:18 } );
src\tests\fixt.rs:16:15: 18:10 help: run rustc --explain E0308 to see a detailed explanation
error: aborting due to 6 previous errors
Could not compile expressions.

16

Take a look at the book on taking closures as arguments.

I think what you're after is something like this:

fn Setup<F>( tf: F ) where F: Fn (&TestFixture) {
    let o = TestFixture { name: String::from ("Initialized") };
    tf (&o);
}

So you can pass in a borrowed fixture, do you standard setup, and then pass it to your closer to execute.

That should hopefully work :smile:

1 Like
17

KodrAus, thanks for that. This is my solution based on your fix:

#[cfg(test)]
mod fixt {

    struct TestFixture {
        pub name : String
    }

     fn setup<F>( tf: F ) where F: Fn (&TestFixture) {
         let o = TestFixture { name: String::from ("Initialised") };
         tf (&o);
     }

    #[test]
    fn make_sure_foo_works() {
        setup(| x | {
            assert_eq!(x.name, "Initialised");
        } );

        // Do a similar thing here with a teardown although need
        // to figure out how to get hold of the orginal TestFixture
        // to perform cleanup on it.
    }
}

This now works. I've not done that much rust yet, which is why I've stumbled on many of the issues. I've read most of the rust book, but it hasn't all sunk in yet. I just needed something like this so I can start to write some meaningful tests. Thanks for the leg up :slight_smile:

4 Likes
18

No problem! You're actually already using quite a lot of Rust syntax in your example test fixture, so there aren't too many more surprises in syntax. The rest is borrow checking :slight_smile:

19

To bring closure to this question, I am posting a complete solution to the problem of not having fixtures. Also, the naming of the variables in my previous post was not particluarly helpful, in actual fact, I can see how it could be confusing to the reader, so I decided the re-write it for clarity. The solution I propose requires the use of a macro as you will see (inspired by the test! macro as defined in cargo). To paraphrase, what I wanted was the ability to define a test fixture with both setup and teardown methods (a la xUnit), which I could use to enforce principle of DRY (don't repeat yourself).

struct FooTestFixture {
    pub name : String
}

impl FooTestFixture {
    fn setup() -> FooTestFixture {
        FooTestFixture { name: String::from("Initialised") }
    }
}

fn teardown(fixture : &mut FooTestFixture) {
    fixture.name = "".to_string();
}

#[test]
fn heap_foo_fixture_should_be_initialised() {
    let mut fixture : FooTestFixture = FooTestFixture::setup();

    assert_eq!(fixture.name, "Initialised");

    teardown(&mut fixture);
}

macro_rules! unit_test {
    ($name:ident $fixt:ident $expr:expr) => (
        #[test]
        fn $name() {
            let mut $fixt : FooTestFixture = FooTestFixture::setup();
            $expr;

            teardown(&mut $fixt);
        }
    )
}

unit_test! (heap_foo_fixture_should_be_initialised_using_macro f {
    assert_eq!(f.name, "Initialised");
});

The unit_test! macro $name is the name of the test, $fixt refers the binding that represents the test fixture and $expr is the body of the test. As it stands, this macro would need to be defined for every test module with the same test fixture, and some might say this limits it's use, but it does serve the purpose. Logically, to get round this the macro would need to be generic (but I don't think there is such a feature in Rust!). I just discovered that I maybe able to abstract out the name of the fixture type (in this case FooTestFixture) by introducing a ty parameter to the macro ....

20

Ok, so this is the end game; I've changed the macro to include an additional identifier representing the type of the fixture, so it can be used accross different test fixtures:

macro_rules! unit_test {
    ($name:ident $fixt:ident $ftype:ident $expr:expr) => (
        #[test]
        fn $name() {
            let mut $fixt = $ftype::setup();
            $expr;

            teardown(&mut $fixt);
        }
    )
}

unit_test! (foo_fixture_should_be_initialised_using_generic_macro f FooTestFixture {
    assert_eq!(f.name, "Initialised");
});

Done!

1 Like