API design question - writing "adapters" for testing

Hi,

I've been recently working on a simple HTTP API client, but on a higher level than libraries like Reqwest, ie. a client that knows about business domain. Let's say that we have a blog and it returns a list of posts when requesting /posts.

An API Client could look sth like that:


#[derive(Deserialize)]
struct Post {
  title: String,
  text: String,
}

struct APIClient {
  client: reqwest::Client
}

// we need to define a `new` method that creates the client, but I'll skip this

#[derive(Deserialize)]
struct PostsResponse {
    posts: Vec<Post>,
}

impl APIClient {
    async fn fetch_posts(&self) -> anyhow::Result<Vec<Post>> {
        let response = self.client
            .get("https://blog.example.org/posts")
            .send().await?
            .json::<PostsResponse>().await?;

        response.posts
    }
}

This works OK, but usually in such situations there is a problem of testing other code that relies on the APIClient. For example let's say you have a function getting all of the titles:

async fn get_titles(&client: APIClient) -> Vec<String> {
    client.fetch_posts().into_iter().map(|p| p.title.clone() ).collect()
}

Now in order to test this I could use one of the mocking libraries and mock the fetch_posts method, but how I usually solved this problem in the past in other languages is to create different adapters. We can create a trait like so:

#[async_trait]
trait APIClientBehaviour {
    async fn fetch_posts(&self) -> anyhow::Result<Vec<Post>>;
}

Then we could do impl APIClientBehaviour for APIClient and implement the fetch_posts method there.

Now with all this setup we could also create a test client that could be used instead of the real client:

struct TestClient {
    self.fetch_posts_response: Option<anyhow::Result<Vec<Post>>>,
}

#[async_trait]
impl APIClientBehaviour for TestClient {
  async fn fetch_posts() -> anyhow::Result<Vec<Post>> {
    match self.fetch_posts_response {
      Some(response) => response,
      None => Ok(vec![Post { title: String::from("title"), text: String::from("text") }]),
    }
  }
}

So if we set a response for fetch_posts on the test client, it will return the response and if not, it will return a default.

Now the problem is this code won't compile, because you can't move the test response out of the TestClient struct. There are two ways that I can think of to fix it:

  1. Change the signature of the function to use &mut self. Then sth like this would be doable: let mut response = None; std::mem::swap(&mut response, &mut self.fetch_posts_response);, which would clear the test response and return it. Depending on the intention it might be OK to do it. The problem with it is that this is needed only for testing. There is no need to use &mut self for the actual code.
  2. Add #[derive(Clone)] to Post and add clone() when returning the response. This will also work just fine, but then we will always need to ensure that anything returned is cloneable. Which is easy to break. In the examples here I used anyhow, but in my actual app I was defining some errors using the thiserror library and one of the errors included the source error of a reqwest::Error type, which is not cloneable

So I think that given those two options I would go with the first one, cause the second wouldn't work in my specific use case, but I was wondering: is there any pattern in Rust world that would help me here? How do you solve such problems? How do you mock API clients? Any suggestions welcomed!

I would be inclined to think that a mock client can make up data on the spot and thus it can construct the return value just-in-time, rather than having to move out of itself.

However, if that is not the situation, I would first try cloning – I don't completely understand, but are you also returning errors in the Ok case? Or how exactly did you end up needing to clone an error? I'm curious because in most cases, responses must also be Deserialize, which non-cloneable errors aren't (since the reason they aren't is usually that they contain a dyn Error somewhere deep down). I would recommend making all your responses into pure data structures, which don't carry any non-trivial information or behavior, anyway – this will likely make them cloneable too.

If you still want to move out of a reference while retaining the immutable &self in the signature, you can just put a RefCell around your mock data, and then you will be able to borrow it mutably, even when behind an immutable reference.

I usually don't. My API client design usually boils down to:

  • a trait Request: Serialize, which is nothing more than a dumb data structure, possibly with methods/constants for defining the endpoint, the HTTP verb, and/or the headers.
  • An associated type Response: Deserialize on the request trait, which likewise denotes plain old data.
  • A Client type with a single method:
    impl Client {
        fn send<R: Request>(&self, request: R) -> Result<R::Response> {
            …
        }
    }
    
    This does nothing more than spinning up an HTTP request, set its URL, method, and serialized body, then receives the raw response and deserializes it back to a domain type.

In my view, there's nothing non-trivial I could test in this whole chain of actions, since it is simply the composition of features provided by other libraries (e.g. reqwest and serde), and the client really doesn't do anything other than serializing and parsing HTTP data.

If anything, I could test that the HTTP requests are sent to the correct place and that whatever the server returns is represented faithfully in the deserialized response. For that, however, I wouldn't need a mock client – I would need a real HTTP client pointing to the test environment of the backend, and some predictable test data on the backend side.

1 Like

In my app the response is of a type Result<SomeStructOrVecOrSth, Error> (where Error is a custom error type. I'm using thiserror and one of the options is:

RequestError(#[from] reqwest::Error)

With that in mind the response can be either an Ok or an Err, so both need to be clonable.

Sorry, the question I asked there was incorrect, what I wanted to say there is "how do you mock your API clients?". And then again it's not about simple API clients, but clients that map responses into domain object. So I'm mainly after how to test code that depends on the API client. Cause even if this code is relatively simple (like: fetch stuff, put into a DB), I would still prefer for it to be tested.

I'm not sure what you mean by that. Let's say that I have a method that fetches the posts and puts them into a database with a signature: fetch_and_save_posts(&client: APIClient). Now imagine you want to test it:

let client = TestClient::new();
let post = Post { title: String::from("title"), text: String::from("text") };
client.fetch_posts_response = Ok(vec![post]);

 fetch_and_save_posts(&client);

// assert that the post has been inserted to the DB

That's what I mean when I say that I'd like to test code that depends on the APIClient. Do you see any other good way to do that?

The design I sketched out above does map responses into domain objects.

Oh, I see. In that case, you probably shouldn't involve the API client at all. Instead, design the other part of the domain logic in a way that it doesn't depend directly on the client, only on the data that you would eventually get out of the client, and then you should be able to create the required input data in the test, rather than relying on a mocked API client.

In addition, if you want to enter paranoid mode, create separate end-to-end tests (using real HTTP round-trips) as well, which only test the system in a coarse-grained manner, but they exercise every part and all relevant interactions of the system as if during real usage.

That's a good separation of different levels of abstraction, but this is already what I'm doing, usually using some kind of a repository pattern or sth like that. The issue is, you still need to fetch the data somehow. I'm not sure if you've missed my previous post here, but there is a following example there:

let client = TestClient::new();
let post = Post { title: String::from("title"), text: String::from("text") };
client.fetch_posts_response = Ok(vec![post]);

 fetch_and_save_posts(&client);

// assert that the post has been inserted to the DB

I think this is a fair thing to do. The happy path in there might be very simple, but sometimes you need to do a bit more stuff in case of errors. Maybe retry? Maybe schedule a job to try later? Maybe do something else? Thus I would prefer to test it and ideally I would test it without mocking full http responses.

So basically, you are not testing DB insertion – are you trying to test that the client can handle errors beyond your control? What errors, other than HTTP errors (which you can't reasonably test with a mock client), should an HTTP client anticipate and handle?

1 Like

I think it's more about testing the integration between the API client and the code that's persisting the data, but now that I read your questions I think I might be overcomplicating. I'm used to split this kind of code into different layers and testing it on different levels, but this is because testing in this way in dynamic languages is relatively cheap. In case of Rust, where it may be a bit involved, but also where the type system is checking a lot of the stuff that you have to test for in dynamic languages, it might indeed not be worth an effort

Thanks a lot for this discussion, it cleared some of my habits from other languages I think :sweat_smile:

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.