Testing against a data file and pass/fail on every entry

help
1

I'm writing a conformance test in the form of rust integration tests which reads some data from a file (four fields per test case) and calls into the public API of the lib crate and compares the result.

Now, the problem is that I'm checking all the test case from my source data file in one #[test] function, which results in failing the test on the very first test case failure. I want to simulate a #[test] fn situation for each single test case, so we can start improving the number of failures procedurally, instead of having to fix the failing test case that's ordered first.

I have considered using a macro, but since the data file needs some more-than-obvious text processing, it doesn't look like a suitable solution.

I have searched many of the resources online and couldn't find any documentation on the internals of rust's test platform (have not digged into rust source code, yet, though), and couldn't find anything helpful.

:question: So, is it possible at all? If not, is there any specific reason for it, or just lack of public interest?

2

Let me add that I've seen code with panic::catch_unwind() calls that handle failures (like this), but that's not what I'm looking for, as after catching the failures (via catch_unwind or just in a loop), I still want to be able to tell the test framework which cases passed and which ones failed.

3

You could write a non-test function in your test that loads the file and presents a single case, like fn get_case(sourceFile: &Path, case_num: u32);. Sort of a helper function. Then use a macro where the invocation chooses which number of case to get, and maybe adds a suffix for that test case to the test function name.

4

Thanks, @ryan. So, now that you suggested that, I couldn't figure out how to concat a function name prefix (like test_) to a macro ident. Is that possible already?

Also, do you mean that we don't have access to the internals for test framework?

5

You know, I don't think you can concat the name. I've only had need for a few macros, but I've never tried that.

Here, look at this as an example: macro_rules from rust by example.. What you're looking for should be near the top of the page.

This whole 'Rust by Example' has been a great resource.

EDIT: This is half the solution, you can pass the function name and the index (as an expression) to the macro, and have it parse both.

1 Like
6

Hum... let's assume that I don't want to parse the test file on compile time. In that case, there's no way I can use macros to create N test cases on compile time (annotated with #[test]) to be executed by the test framework.

(I'm studying to see how much would it be if I want to parse my data file on compile time. Here's my starting point: https://www.reddit.com/r/rust/comments/2aur8x/using_macros_to_parse_file_formats/)

Now, what would be the run-time solution here?

Here's a simplified version of what I have right now: Rust Playground

What I want to do is to unmark my_tests() from #[test] and execute every single call to my_test_case() as a #[test] call.

7

As far as I know, Rust doesn't have compile time evaluation (at least not in stable) yet, so you would have to use a build script to do something like that.

So I'm assuming you would write a test module that looks something like this.

mod test {
    macro_rules! make_test {  
        // macro logic
    }

    // Helper function
    fn load_file_and_get_case(case_num: u32) -> SomeDataTypeForACase {...}

    // Macro invocations
    make_test!(test1, 1);
    make_test!(test2, 2);
    make_test!(test3, 3);
    make_test!(test4, 4);
    make_test!(test5, 5);
}

where a macro_invocation would expand to something like this for make_test!(test1, 1);

#[test]
fn test1() {
    let data = load_file_and_get_case(1);
    /// asserts below!
}

In the link I provided above, it shows how to write the macro to create a function. I imagine this could be pretty slow if you have a lot of tests, because it would open/close the file for every case!

EDIT: If you don't want to hand type all the macro invocations, you can write another macro that does that for you, so you can just type in a list of test case names and numbers.

8

Okay, I found the code in rustc source (rust/src/libtest) and realized that these are called Dynamic Tests, which are non-static functions.

The problem, though, is this crate (called test) is not stable and shall not be used in production code.

But, @SimonSapin has created a stable fork of it that can be used instead. Read more about that and the price to pay (for now, until test is stable) here: #[test] and external test harnesses - language design - Rust Internals