Is there a way to mark all tests in a file/module as `#ignore`?

help
1

Hi all! Relative Rust newbie here.

I'm writing some integration tests for my Rust project. It depends on certain services being up and able to respond, so it shouldn't be run if those services are not available. So, I want to use cargo test to run them, but I don't want them to run by default, because they're not unit tests and I don't want people to accidentally try to run longer tests. I've been using the #[ignore] attribute for this, which works a treat. I can write helper scripts that set up and run the services and the integration test, and I can run these in my CI system, but I don't accidentally run these tests every time I run cargo test.

What I'd like is to have a way to mark a set of tests with the #[ignore] attribute, rather than having to mark each test that way. That way I don't have to remember to add it to each of these integration tests. It's a minor inconvenience to have to type it, and I catch it relatively quickly when I don't, but I still think it would be useful. But I can't seem to find a way to do it. I tried putting all of my test functions under a mod test and adding #[ignore] before the mod, but that didn't seem to have an effect. Is there an easy solution I'm missing? Thanks.

2

We've got a similar situation at work where the integration test suite may take 5+ minutes and you won't necessarily want to be re-running them every time you hit save.

My solution was to move the integration tests into their own crate and use cargo aliases so cargo integration-tests and cargo it are shorthand for cargo run --binary integration-tests.

3

I've used features before, to disable specific tests when the feature is enabled, not strictly additive but whatever :wilted_flower: using something like:
#[cfg_attr(all(test, feature = "ignore_kfail_tests"), ignore)],

You can conditionally compile the tests out, but it has the negative impact of not actually showing those tests as being ignored in the test summary.

#[cfg(all(test, not(feature = "ignore_foo")))]
mod test {
}
2 Likes
4

I saw the suggestion of features around other questions about long-running integration tests. It's a decent idea, but I like the simplicity of the ignore paradigm, and I was hoping I could extend it this way. But I might switch to the feature approach, depending on how annoying this gets.

5

Interesting--what does the binary look like? Do you still mark your functions as #[test] or is it running a custom test program?

6

We use a custom test harness because our tool is essentially a compiler. I copied rustc's system where you have folders like compile-pass, compile-fail, run-pass, and run-fail and in each folder you have some programs which should compile/execute successfully or unsuccessfully, with additional files so we can make assertions based on the output (e.g. stderr.contains("Error: version number must be specified")).

1 Like
7
TL,DR

This is the approach I'd go for; except for --cfg test being included there, since the cfg-test-gating itself already ought to happen at the module level if unit-testing, or inherently if integration-testing:

//! If unit-testing.
#[cfg(test)]
mod special_integration_tests;

//! `src/…/special_integration_tests.rs` if unit-testing
//! `tests/special_integration_tests.rs`

#[cfg_attr(not(feature = "special-integration-tests"),
    ignore,
)]
#[test]
fn …

Granted, it's a bit of a mouthful, and in that case writing a helper macro_rules! for this can be quite handy;

macro_rules! special_integration_test {( $fn:item ) => (
    #[cfg_attr(not(feature = "special-integration-tests"),
        ignore,
    )]
    #[test]
    $fn
)} pub(in crate) use special_integration_test;

so that:

special_integration_test! {
    fn …
}

Just Works™.

Should the incurred rightward drift be deemed ugly / unergonomic, there is the ::macro_rules_attribute crate to help with that:

use ::macro_rules_attribute::apply;

macro_rules! special_integration_test {( $fn:item ) => (
    #[test]
    #[cfg_attr(not(feature = "special-integration-tests"),
        ignore,
    )]
    $fn
)} pub(in crate) use special_integration_test;


/// This is ignored unless `--feature special-integration-test` is added to `cargo test`
#[apply(special_integration_test!)]
fn …
2 Likes
8

Yeah, in my case I am using a custom test harness as well, so I only have one of those cfg_attr things which I stuff all my kfails into, and so adding extra stuff to make it nicer just adds more stuff.

Edit:
Similarly you flipped the context, e.g. I want to run these by default locally but be able to ignore them for CI.
Both are definitely valid options depending on what you want to do

1 Like
9

Thanks! I haven't dove into actually writing macros yet. I kind of guessed that that might be a solution, but I know absolutely nothing about the macro syntax.

10

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.