Is it just me or mockall is not well supported by IDEs?

Hi all, started learn rust few weeks ago and now I am in a state where I writing some code.

I am using mockall as mocking library and currently I am trying to mock simple Structs as explained in docs (mockall - Rust) From compilers point of view it's all good and works however today I realized that both VSCode and Intellij started to when I use mockall.

Example #1 mockall_double::double breaks Intellij. Intellij can't resolve functions so autocomplete and GoTo definition etc doesn't work;

Screen Recording 2023-06-10 at 21.30.20

I just wanted to check if it's just me or it's bug in IDEs so I should report it in respective places.

Thanks!

I can't put two links in one post because I am a new user so here's the other one :slight_smile:

Example #2 automock breaks VSCode autocomplete:

Screen Recording 2023-06-10 at 21.36.15

I can't speak to your specific IDEs or the mockall crate, but in general, macros and particularly proc macros and especially attribute proc macros are difficult for IDEs.

1 Like

This was sad to read but thanks for sharing anyway!

Maybe that’s the reason why almost all the rust programs I have seen don’t have tests :neutral_face:

Most popular libraries I've looked at (the source of) have them.

I wonder what programs are you talking about. The ones I've seen have enough tests, usually.

Sure, they don't have mocks and/or unittests, but that's normal: if your language doesn't have an implementation inheritance then unittests and mocks are something you should need excessively rarely, it's mostly a sign of either bad design or something very tricky… usually you want to avoid both, if you can.

1 Like

Maybe that’s the reason why almost all the rust programs I have seen don’t have tests

To clarify, I am looking at mostly backend examples and I wasn't referring to libraries. I must admit that this was an over-statement.

if your language doesn't have an implementation inheritance then unittests and mocks are something you should need excessively rarely, it's mostly a sign of either bad design or something very tricky… usually you want to avoid both, if you can.

I have this simple use case;

pub struct UserRepository {
    pool: Pool<AsyncDieselConnectionManager<AsyncPgConnection>>,
}

impl UserRepository {
    pub async fn insert_user(&self, new_user: NewUser) -> Result<UserWithId, AppError> {
       todo!();
    }
}


pub struct UserService {
        pub user_repository: UserRepository,
}

impl UserService {
    pub async fn register_user(&self, register_user_request: RegisterUserRequest) {
   todo!();
   }
}

There will be a business logic in service layer that I want to test and I don't want to depend on databases running. Mockall support mocking this Structs but IDE becomes unusable.

The other alternative I know is using traits (just for the sake of testing - not very cool but it's ok) and generic type parameters to avoid dynamic dispatching (not sure if this is a real problem but I have been told it's bad signal (?) if you need to use Box<dyn YourTrait> etc).


trait UserRepositoryTrait {
    pub async fn insert_user(&self, new_user: NewUser) -> Result<UserWithId, AppError>;
}

pub struct UserRepository {
    pool: Pool<AsyncDieselConnectionManager<AsyncPgConnection>>,
}

impl UserRepositoryTrait for  UserRepository {
    pub async fn insert_user(&self, new_user: NewUser) -> Result<UserWithId, AppError> {
       todo!();
    }
}

trait UserServiceTrait {
    pub async fn register_user(&self, register_user_request: RegisterUserRequest);
}

pub struct UserService<T: UserRepositoryTrait> {
        pub user_repository: T,
}

impl UserServiceTrait for UserService {
    pub async fn register_user(&self, register_user_request: RegisterUserRequest) {
   todo!();
   }
}

Ideally with this approach one can choose to not use mock library and create it's own dummy implementations. Dummy implementations I believe in theory makes sense but in practices they can get very verbose (especially if your layer has many functions etc) so I would prefer to use mock just to simply mock few calls without abusing the mocks.

All that being said, I haven't try to use mockall with trait approach and check whether it breaks anything. However, issue seems to be related with macros so I am not so hopeful.

Please let me know if you know other ways to test Rust program :slight_smile:

Thanks in advance!

Former requirements makes sense to me, latter… not so much. One can easily create in-memory database for tests if you don't want your tests to hit the disk.

Mocks always, 100% of time mean something is badly designed somewhere. Because you are testing something that, ideally, you shouldn't care about at all. Why would you care which exact requests to SQL database would Diesel use and when?

The only justification for mocks is when bad design is something you can not control. Because it's part of OS, or it's embedded in some existing protocol, etc.

In these cases it's Ok to note to herself “Ok, I'm dealign with something where redesign is impossible, let's use mock” and do just that, if you need mocks because of something you designed… just don't do that.

1 Like

Why would you care which exact requests to SQL database would Diesel use and when?

Isn't this the whole idea behind the mocking (+ interfaces)? I have a repository it could be backed by filesystem, it could backed by database or it could be backed by API. The point is I don't care just give Result<Option, AppError> and I would like to test the behavior of my business logic when I got None, or Some or different AppError. I don't need to fiddle around to simulation that exact behavior to get the result I need.

Long story short, I don't seem to agree with your view on mocks but that's ok :slight_smile: Mocks always, 100% of time mean something is badly designed somewhere

It feels like you more like preferring integration test or black-box test (however you want to call it). Today, I was looking at lemmy which is written in Rust and that's exactly what they have. Bunch of high level tests from API point of view lemmy/api_tests at main · LemmyNet/lemmy · GitHub.

I do personally still believe being able to isolate/mock external dependencies (be your internal code or be it external - doesn't matter) in unit test is still valuable.

Regardless, thanks for taking time to share your point of view :slight_smile:

Obviously.

So take one of these normal implementations and use them in tests. Why do you need mocks? What's the point? What are you trying to achieve by mocking?

Sure. That's why mockall exist in the first place. But if you feel the need to use it then it means that API that you are using is unsound. It happens. Especially when third-party code is concerned.

But most of the time the solution is not mocks but sound wrapper.

Rather than utilize mocks, Rust's system benefits from a "sans IO" style; instead of business logic talking to the database directly, the business logic (attempts to mostly) functions without talking directly to IO, taking a more pure computational shape. Then there's a small glue/integration layer that should be fairly trivial and not need standalone "unit" testing, just the "integration" testing that you wouldn't use mocks for anyway.

Ofc "you don't need mocks if you just don't write overly coupled code" is a bit flippant, but Rust really does prefer if you and help you to couple components less tightly.

It's not always as simple as that, of course; if the business logic needs to request further IO, or you want to work in a streaming fashion, business logic can't just be a pure function. But that's what the IO traits (and other similarly shaped traits) are for; you can fairly easily swap between your production IO and a simple in-memory provider. The difference to mocks is that you're still providing a (mostly) complete implementor of the interface, just a simpler one that doesn't need to handle all the annoying realities. Traits are the way to swap between providers, as the equivalent of and doing the same thing as how you'd abuse inheritance to provide RepoMock where Repo was originally expected.

I quite like matklad's perspective in How to Test, which comes from working on rust-analyzer, which is a perfect case study of both difficult to test (giant persistent ball of state) and simple to test (well defined oneshot queries over that ball of state).

The biggest takeaway imho is that rather than mocks (expect these interface calls to happen and return these values) you typically do want to have your tests using a "real" but fast/local provider. Ideally, all of your tests should work the same and continue to pass if the component being tested were to be ripped out and replaced with an AI black box. That's an aggressive way to put it, but it's a good way to ensure that the tests aren't getting in the way of should-be-encapsulated refactors.

In fact you can argue that mocks increase coupling, since you're now testing the exact way the mocked component gets used[1], rather than just that the result is correct. At a minimum, it increases the friction of refactoring how either component talks to the other, since the mock tests are tightly coupled to both.

If you already have that dynamicism between those implementations, then that dynamicism is the exact place where you would provide

Creating a new trait solely to mock a component can be a minor code smell, but it's so because it's relatively likely this isn't a proper place to be injecting a mock-like, if there's only one possible actual implementation. If it's a layer that it makes sense to have alternative providers for, on the other hand, the generality makes sense and it's not really a code smell.

It's absolutely fine to use Box<dyn Trait> for this kind of looseish coupling boundary. In your traditional OOPish system where mocking is a simple abuse of inheritance away, it's almost exactly equivalent behavior to what everything always does. The important part is that whenever you have dyn Trait that it's a looseish coupling and you actually do function with whatever correct implementation of the trait interface, and aren't relying on the specific behavior of some concrete implementation behind the dynamicism.

Static dispatch is "better" for performance in theory (but worse for compile time and code size), but only marginally in most cases. There are some things you can't do with dyn, but if your interface is dyn-compatible, it's almost certainly fine to use trait objects.

The main reason people shy away from them in public interfaces is that you can typically provide a trait object where a generic is asked for, so that leaves the choice to the caller. But being dynamic is much simpler and typically nicer to compile times as well.

The main thing is isolation. This is especially important since cargo multithreads tests in parallel by default, so it's easy for tests to interfere with each other and slow each other down, even if they don't cross-pollute results.

Many would probably call what rust-analyzer's tests do as mocking; they set up essentially an entire virtual file system for each test that they then ask the inference engine to look at.

But the key insight is that it isn't a mock, though; it's a fully functional provider. But it provides the same isolation benefits that a mock is designed to. Just without encoding an expectation of how exactly the provider is consumed.


  1. Um, actually, :nerd_face::point_up_2: there are cases where mocks really are what you want: when you're testing a transit layer, you want to test what gets through. That's essentially what serde_test is doing (serializing makes these calls on the serializer, these deserializer calls work to deserialize), and tracing has a mock layer to test that event filtering etc. is working as expected. But notably, both of these are one-sided mocking which serve just as a sink or a driver, rather than a call/produce interaction. The two take the two major different approaches, as well; serde_test records the entire process to replay and assert against afterwards, where tracing-mock takes a prior description of what's expected and asserts as it tracks down the playbook. The former is nicer for seeing the big-picture, but the latter produces more useful backtraces for where things diverged from expected. ↩︎

1 Like

That's not quite what unsound means. Unsound specifically means that it's possible to cause UB using safe code, where UB has a very precise definition as an operation outright forbidden and assumed to never happen by the Rust abstract machine.

Mocking/isolation from dependencies isn't about isolation like process isolation to control the blast radius of UB. It's much more simple: being able to test your code layer without also testing the downstream code (which is assumed to be correct).

Typically the desire for isolation comes from wanting to take some shortcuts over the full-fat dependency to speed up testing. It's exactly the same use case as swapping between any IO providers, just that the motivation comes from "make my tests faster and/or rely less on the environment" rather than some production desire.

That's precisely my point. Mocks enforce certain contract between your “business logic” and other layers which is way, way, WAY beyond what's needed and/or desired.

Yes. And instead of making tests rely on the environment they now rely on minutiae details of your implementations. Your tests don't test the business logic now, instead they test how that business logic is implemented.

I've seen that pattern many, many, MANY times: code is fully covered by tests, there are thousands upon thousands of tests… and yet when requirements change — bugs are found by humans testers, not by all these testsuites.

Why? Because tests should not just break when something is wrong, but, equally importantly, they have to not break when something is not wrong. False positives and false negatives. Mocks fail that idea in extremely spectacular fashion: almost every time if you have to change something in the implementation you need to adjust some mocks, too. Because now “real provider” is different are mocks are the same.

Pretty soon this trains people to automatically adjust mock-based tests when they are changing code: “oh, now function foo is not called and bar is called instead because we need to pass additional parameter… Ok, just need change these 20 or 100 mocks… done”.

And when it reaches that stage mock tests don't test anything except patience of developers who are blindly adjusting them.

Most of the time you either need real provider or simplified, but still fully-functional provider (e.g. in-memory SQLite instead of PostgreSQL). Sometimes you want to add proxy layer on top of real provider (e.g. to inject random network failures or some fuzzing).

Mocks are not the #1 way to test things, rather mocks have to avoided if at all possible. You use mocks when you have to, not when you want to. E.g. if you are writing program for API which is expensive to use (GPT-4?) and you don't want to pay too much for testing or want to test you code without network access.

Mock tests are worst types of tests — but sometimes the only alternative is no tests at all. That is why they exist.

Yet, for some reason, I have seen them abused so many times… yes, they are easy to write and you can achieve amazing coverage which pleases management… but is it enough reason to use them?

Hi @khimru and @CAD97 thanks for lovely writings. I already took some notes to think about.

I think I will try to change my approach and try to stay away from mocks and see I can provide alternative implementation that I can use in test. Cheers.

With regards to the original question, most of proc macro support in Intellij Rust is gated behind a hidden experimental feature. Open "Search actions" menu (Ctrl + Shift + A on default Windows keybindings, "Search Everywhere" also works). Type "Experimental features" and open it. You'll get a list of various enabled and disabled features. Enable all features starting with org.rust.proc. Also, while we're here, consider enabling org.rust.cargo.evaluate.build.scripts. This is likely to fix most issues with non-working proc macros.

Why this feature is still disabled and hidden after many years, I do not know. Of course, there's always a possibility of more complex analysis introducing new bugs.

3 Likes

Hey thanks for the heads-up but I already have all of them enabled. I even tried to disable some of them to see if it makes any difference. I should have mentioned this in the first message sorry.

Regarding the original question, IDEs must choose whether to expand the code with #[cfg(test)] or without it. Most of them expand with it so they can auto complete functions used only by your unit tests. But if you're using Mockall, your code will expand very differently in those two configurations. You may be able to configure the IDE not to set #[cfg(test)] which will make the autocompletions more useful for the production code at the expense of the unit tests. But you have to choose one or the other.

And regarding the more philosophical question of to mock or not to mock, I'd like to share my opinion. Yes, @khimru is correct that using mocks increases the maintenance burden of your test suite. And he's also right that mocks aren't necessary if you can achieve full coverage via black box testing alone. But often you can't. So mocks are very useful when:

  • You need to replace some runtime dependency, like the aforementioned database, GPT-4, etc etc.
  • You need to test error handling, and there's no way to inject errors with a black box approach.
  • You need to inject errors that possibly could be injected with a black box, but doing slow would require exotic operating system features, be slow, need root privileges, etc.
  • You need to inject super-weird errors like hash collisions, or even triple hash collisions.
  • You need precise control over the timing of certain events, like the order in which Futures complete.
  • A black box test would suffice to test correctness, but you also care about performance. For example, exactly how many database queries will the foo method make?
  • You need to test a certain operation across a range of internal state conditions. A black box could theoretically generate all such conditions, but it would be hard to do, and very difficult to verify. In this case, using mocks might actually be less maintenance-intensive than not. One example would be operations that mutate a complicated data structure, like a B-Tree.

I'm sure there are plenty of other cases, too. So my usual approach to testing is "all of the above". When I design a project that has multiple layers, I usually write unit tests for each layer using a mock version of the next lower layer. Then, I also write functional tests for each layer using the real version of the lower layer. Finally, I write integration tests for the top layer, the user-accessible one. Then for each test case, I write it in whichever location seems more natural. That usually means an integration test if possible, a functional test if not, and a unit test if I absolutely must.
I'd also like to draw a distinction between mocks (programmed with a "expect foo then return bar" interface) and fakes. The latter are a somewhat functional version of the real thing. For example, an in-memory database instead of an on-disk one. They're what @CAD97 was talking about. Fakes are useful too. But if you don't have a ready implementation, then creating one could be a lot more work than using mocks. So I don't personally use them very often.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.