Hi, My name is Erik and I just don't know how to test

Like the title, I have been developing for practically all my life, but I never learned how to write proper tests.

I of course worked for companies where we had to write tests, and so I did. For example, I wrote tests to verify that the given property return type was indeed an integer and not a string. And what the program would do if it was not an int. This was in the Python ecosystem. Where type juggling unfortunately is a thing, and typecasting is merely a visual indicator.

These tests were of course pretty simple to write, extremely mundane, uninspiring and honestly just annoying to do. I rather use my time to solve issues, optimize and implement instead of figuring out HOW to write a test for it, and which libraries I need to use to mock X or Y and get that 100% tested mark on all lines of code even though it felt so redundant for certain functions/models.

Therefor, with all the experience I gathered in my life, I still feel like I am stuck on this and honestly, feel embarrassed about it too.

I switched to Rust about 3 months ago, and I have rewritten a big chunk of one of my libraries already. Though, without tests. Today when I started working I sat down and felt the dread to just bite the bullet and start to write/learn tests and already faced a wall.

Example one.

pub fn get_listen_key(
    api_auth: &ApiAuth,
    test_net: bool,
) -> Result<ListenKey, BinanceConnectError> {
    // Create a new HTTP client.
    let client: Client = Client::new();
    // Determine the appropriate Binance base URL based on the test_net flag.
    let endpoint: String = format!("{}{}", base_url(test_net), constants::FUTURES_LISTEN_KEY);
    // Send a POST request to obtain a listen key.
    let response: Response = client
        .post(endpoint)
        .header("X-MBX-APIKEY", &api_auth.api_key)
        .send()?;

    // Check if the response status is OK (200).
    if response.status() == StatusCode::OK {
        // Deserialize the response JSON into a ListenKey struct.
        serde_json::from_str(&response.text().unwrap()).map_err(BinanceConnectError::JsonError)
    } else {
        // Handle non-OK HTTP status codes by returning an error.
        Err(BinanceConnectError::HttpResponseError(format!(
            "Not-OK status code received {:?}",
            response.status()
        )))
    }
}

This function is extremely straightforward. It does a POST rest call on the Binance listen_key url. The arguments are an ApiAuth struct holding the api credentials and a flag to use test_net or not. On a 200 it will serialize the result to a ListenKey struct and return, on an error it will propagate the error via een BinanceConnectError.

The only thing I can write tests for (disregarding that I dont know how to mock the server), is that it either retrns the ListenKey struct, or the Error struct... I just don't see a way to test this properly.

  1. I can not call the Binance server for every test (also, I don't have api keys in the repository, so it does not make sense).
  2. When I want to add a mock server I also have to create an 3rd argument for the get_listen_key to potentially add the mock server for the test, which pollutes the code a lot. Testing has nothing to do with that function, and it should not know of its existence in my opinion.
  3. If I add the mock server I write explicit statements where I say, the result is now 200, return exactly what I expect, I can not wrap my head around how this is beneficial.

Example two:

pub fn client(
    sender: Sender<Event>,
    url: Url,
    would_block_config: WouldBlockConfig,
    subscribe_payload: Option<String>,
) -> Result<(), BinanceConnectError> {
    // Establish a WebSocket connection.
    let mut socket: WebSocket<MaybeTlsStream<TcpStream>> = socket(url)?;

    // If a subscribe payload is provided, send the subscription request.
    if let Some(subscribe_payload) = subscribe_payload {
        debug!("{:?}", subscribe_payload);
        socket.send(Message::Text(subscribe_payload))?;
    }

    // Continuously read and process WebSocket messages.
    loop {
        match socket.read() {
            Ok(message) => match message {
                // Handle incoming JSON messages.
                Message::Text(json_response) => {
                    // Deserialize the JSON into an `Event` and send it to the sender.
                    let event: Event = deserialize(json_response)?;
                    sender.send(event)?;
                }
                // Handle incoming Ping messages.
                Message::Ping(ping) => {
                    // Respond to Ping with Pong to keep the connection alive.
                    socket.send(Message::Pong(ping))?;
                    debug!("Pong");
                }
                _ => {}
            },
            Err(err) => match err {
                tungstenite::Error::Io(ref io_err) if io_err.kind() == ErrorKind::WouldBlock => {
                    if would_block_config.error_on_block {
                        // Return a SocketError if configured to do so.
                        Err(BinanceConnectError::SocketError(err))?;
                    }
                    // Sleep for the specified time if a WouldBlock error occurs.
                    info!(
                        "futures_usd client thread slept {:?} because of WouldBlock error",
                        would_block_config.time_out
                    );
                    std::thread::sleep(would_block_config.time_out);
                }
                _ => {
                    // Return a SocketError for other types of errors.
                    Err(BinanceConnectError::SocketError(err))?;
                }
            },
        }
    }
}

Basicly the same, I am connecting to a websocket, and expecting certain types of data, though I can not connect to the websocket during a test, therefor I should mock the websocket connection and again need to explicitly write what I expect, therefor I feel defeating the purpose. I am just kinda lost here and maybe have a very different idea in my head about testing than it in reality is.

How would I tackle these tests? Which tools/frameworks do I need to do this properly?

All advice is welcome, any links to documentation or books. I even want to throw money at a course and sit with it for a couple of days.

1 Like

When I want to add a mock server I also have to create an 3rd argument for the get_listen_key

You've already got a parameter that's about controlling the server: test_net. Just make it an enum that permits specifying a substitute. (Or accept the URL directly instead of constructing it; that's good HTTP practice in theory, though might be problematic for your specific case.)

enum Net {
    Production, // or whatever test_net = false means
    Test,
    #[doc(hidden)]
    LocalTest { url: String },
}

pub fn get_listen_key(
    api_auth: &ApiAuth,
    net: Net,
) -> Result<ListenKey, BinanceConnectError> {

If I add the mock server I write explicit statements where I say, the result is now 200, return exactly what I expect, I can not wrap my head around how this is beneficial.

It may not be. Don't assume that every possible test you could write is actually worth writing. Knowing what to test is a skill that one builds as a programmer, and adapts to the current circumstances.

Things that you might consider bothering to test are such situations as:

  • What if the server doesn't return 200, or returns nonsense? Does your client recover from errors well, without panicking?
  • If the websocket connection is dropped by the server, do you reconnect successfully and continue operating correctly?

Don't worry too much about testing the successful cases, because successful cases will often be incidentally tested along with the edge-cases that you should bother testing in order to write robust software.

2 Likes

I liked this blog post about that:

1 Like

I generally recommend simply not writing this particular kind of test.

My opinion is that testing HTTP clients without additional business logic is futile. Such a piece of code usually does the following:

  • it hopefully delegates the raw HTTP request/response handling to a library and you didn't roll your own respone parser, in which case you don't need to test this part;
  • it parses the response body as JSON or whatever the serialization format du jour is, which isn't something you need to test, either (unless it's something in-house – hopefully not);
  • it manages authentication, which is again either a 3rd party library (like an OAuth implementation) or is needed in other parts of the system, so it should be an independently-tested module anyway.

In short, there's very little in there that is your own code's responsibility. It's mostly glue between libraries, and if you are doing error propagation correctly, then it's actually quite hard to get the aforementioned error cases wrong as well.

If anything, you should create end-to-end integration tests which involve an instance of the actual backend service you are going to talk to. Don't mock servers (or anything, for that matter); that will come with its own set of nasty surprises. Deploy a test service with an empty DB and make requests against that, this ensures a clean state and exercising the actual mechanics of the whole system from client to backend.

If you find that a lot of your code needs mocking, then you shoud probably refactor. A couple weeks ago, there was a similar question about testing an HTTP client with some sort of mock "request writer/reader" which was hard to get hold of. My advice to that person was that instead of creating specialized test fixtures that fool code into talking to an HTTP backend, one should instead design components to work against io::{Read, Write} so that they can be completely oblivious of whether the source/sink is an HTTP connection or a plain Vec of bytes in a test runner.

5 Likes

”My RSI and bleeding fingers have hopefully appeased the testing gods and atoned for my previous omissions”.

Exactly how I felt, thanks for the link!

1 Like

Thanks. This is useful.

Thanks for the reply. I think I will stay clear from trying to mock the connections that are not within my responsibility and make an integration test that tests the internal logic of the library.

This thread reminds me of an anecdote that I read on something like HackerNews about 15 years ago. The commenter worked with a colleague who completely misunderstood the point of testing. To paraphrase the tests that were written:

assert!(1 + 1 == 2);

Which is an absurd unit test for an application because "this tests the compiler" [1]. But I think it's probably useful to show the extreme opposite of what you should be testing. You can extrapolate the lesson to include "don't test the libraries you use" because that's kind of the library's responsibility, not yours as the caller.

The tests you should be writing are mostly of the form, "given this input, does my function return the expected output?" Which isn't the right question to ask for any function that does I/O. For those, you can do boundary condition tests, which are of the form "given this condition, do I get the expected error?" And these kinds of tests are monotonous to write (as you mentioned in the OP) and better automated with other approaches [2].


  1. I'm pretty sure this is now a popular urban legend. I've seen it reformulated in different ways, e.g.: A Type of Test - The Daily WTF and I Fixtured Your Test - The Daily WTF and At Least There's Tests - The Daily WTF ↩︎

  2. There's a lot you can do, here. One idea is generating your client code from an API contract like OpenAPI or gRPC. That's more code you don't have to test. ↩︎

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.