I have several unit tests (the #[cfg(test)] mod test { #[test] fn test() ... variety) that cannot be run in the same process. For example, several must each set the the same static OnceLock to distinct values. Is there a way to annotate a test so that it runs as a separate process instead of a thread?
1 Like
Not that I know of. To me this sounds like unit tests are the wrong type of test for what you are trying to test, or that you could enhance your abstraction to avoid your units depending on global state (i.e. by passing the global state—which you would write to the OnceCell during normal execution—explicitly to the unit).
3 Likes
I think cargo-nextest may run the tests in different processes by default https://nexte.st/
That said those tests would fail if they weren't run in nextest which isn't ideal.
1 Like
Yes, things depends on certain global state are notoriously hard to be unit tested. It's one of the main reasons why global variables are considered harmful.
BTW- there was a typo in my original post that I just fixed: OnceCell -> OnceLock. That probably doesn't impact anyone's comments, though.
I will look at cargo-nextest. But I agree that having tests that would fail when run the standard way is not ideal.
I could also wrap the non-test functions that use the static OnceLock so that their wrapper derefs the OnceLock and passes it as an arg instead. But then I can't test that wrapper - although it would consist only of trivial code.
Certainly, global state is often problematic. But sometimes it is the best approach.
I was hoping for something like a #[test(process)] annotation.
It sounds like rusty-fork crate should be what you are looking for.
"Integration tests" (tests in the tests/ directory at the root of the package) are compiled as separate binaries and therefore run as separate processes. This would be a sufficient solution for a single test that needs to be isolated.
If you have many tests that can run sequentially (the global state is resettable), then configuring the integration test target as harness = false and writing your own main() to sequence the tests would handle that.
If you really need multiple processes, then you could have the test binary spawn itself (more portable than fork) with arguments specifying a single test to run. This is sort of recreating nextest's behavior without requiring the user to use a different tool explicitly.
8 Likes
Adding rusty-fork as a dev-dependency seems like the best solution. Thanks @akrauze ! Although it looks a bit kludgey - because it is not integrated into cargo test, and instead has to parse the cargo test command line - and may be out of date or become out of date with respect to newer cargo test options.
I could make these tests integration tests, but that would require exposing things in API that otherwise shouldn't be there. The only thing about these particular tests that suggests they be something other than unit tests is that they set global state. I've already written them as unit tests, and discovered that they fail because they're running in the same process, hence OnceLock::set(...).unwrap() is failing in whichever happens to run second, but they pass when run individually. Otherwise I wouldn't have known that cargo test runs all tests in a single process.
I forgot to mention...
Personally, I would take this approach. Have at least one integration test that runs your entire program, and presumably your program would break if this static OnceLock code is incorrect, so it will be adequately tested. The rest of your tests can then be robust and run efficiently, because they don't interact with global state at all.
1 Like
This is the generalized "sans io" approach, which has benefits for reusability as well as testing. Separating the pure functions (in this case just avoiding access to globals) may not have benefits other than testing in this case, but it's interesting to me how often it proves useful.
It may be the case that using a thin wrapper for accessing the static OnceLock is a better approach in this case. I will consider it further.
Unfortunately, I've used OnceLock many times in this app because it has a structure where it makes perfect sense: initialize a large complex directed graph structure in the main thread, then spawn many threads that read that structure. It will be difficult to thin-wrap many of those OnceLock sites because they are not static. I will be able to test a lot of that structure with integration tests, but there are some edge cases that are very difficult to test adequately using only public API.
Is there a #[cfg(integration-test)] annotation I can use to expose more API during integration testing? I guess I can always add my own cargo feature to do that.
That brings me to another issue I've hit - that functions/types which are private can't be tested outside their module directly. I've already written re-exporter modules with wrapper functions of the form:
#[cfg(test)]
pub mod privates {
pub fn wrapper_around_foo(x: T) { super::foo(x) }
}
when a unit test in some other module needs to call the private function foo. This is getting a bit ugly, but I'd rather have the ugliness exist only during testing, instead of making functions permanently public when they really shouldn't be.
1 Like
If they are not static, why do they need to be changed for testing? Why can't they be created anew within each test?
The only option is to add a feature. Unfortunately, there's no way to declare “enable this feature when running tests”, but you can declare required-features (or cfg inside the test file) to avoid the test failing to compile if the feature isn't enabled.
Well, that's embarrassing. Of course they can. I'm extrapolating my problems needlessly.
So required-features doesn't enable the feature for that target, it just causes the target to be skipped if the feature is not enabled? I don't think I want to do that.
Yes, unfortunately, Cargo currently has no way to express “this feature is necessary for a normal amount of test coverage”. (Activating features for tests/benchmarks · Issue #2911 · rust-lang/cargo · GitHub is the closest issue I could find.)
However, note that you probably should make your CI (or whatever project test tooling you have) use cargo test --all-features anyway to ensure all features are tested, unless you have no features at all.
Simpler alternative, given that nextest seems reliable and wouldn't incur much cost if you had leave:
- just use nextest (
cargo nextest run) - add a
warn!("CAVEAT: <expected to be run in separate process, why, use of cargo nextest at time of writing>)- (assuming you have something like tracing_test to emit events)
Save time, reasonable process, guided fix for some future user.
Note that putting each test in its own binary means that:
- you pay the cost of linking each binary separately -- this can become onerous for larger projects
- your test run effectively becomes single-threaded, because
cargo testruns each binary sequentially
With nextest (which I wrote and am the maintainer of), neither of these compromises are necessary. In fact, addressing this issue was one of the key problem statements behind the design of nextest.
For this and other reasons, several projects have made the decision to no longer support cargo test and only support nextest. That's a decision that projects would need to make individually, of course.
Hilariously late I know, but this sounds like an ideal use-case for Box::leak()
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.