How do you write integration tests that fail early and often?


#1

Hi,

I’m struggling with integration tests and how I can make them fail more reliably. Example:

extern crate hyper;

use hyper::server::{Server, Request, Response};

#[test]
fn test_port_overlap() {

    let port = 9092;
    // Test that starting the server fails if the port is already occupied.
    // Start a dummy server on port 9092 that just returns a hello.
    let dummy_server = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();

    let dummy_server2 = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();
}

This test should fail immediately because we try to bind the second server to the same port. That causes a panic and any panic should IMO fail a test immediately, right?

Instead, this test run just hangs. You don’t know what’s going on, why the test is stuck. The explanation is that creating a new server spawns a new thread that is never terminated - so although my test function has panicked Rust happily continues to run the first server that was started.

How can I tell the Rust test runner that it should shut down all child threads spawned by a test function when a panic occurs? I probably need some test framework for that?


#2

Tried Stainless, but no luck:

#![feature(plugin)]
#![cfg_attr(test, plugin(stainless))]

extern crate hyper;

use hyper::server::{Server, Request, Response};

describe! stainless {
    before_each {
        let port = 9092;
        // Test that starting the server fails if the port is already occupied.
        // Start a dummy server on port 9091 that just returns a hello.
        let mut dummy_server = Server::http("127.0.0.1:".to_string() + &port.to_string())
            .unwrap()
            .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
            .unwrap();
    }

    it "port overlap" {
        let dummy_server2 = Server::http("127.0.0.1:".to_string() + &port.to_string())
            .unwrap()
            .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
            .unwrap();
    }

    after_each {
        dummy_server.close();
    }
}

Same as the example above - the testrun will just hang but it should fail the test because there is a panic.

Is there a test framework that supports catching panics?


#3

The next approach I will try is to write my own panic handler like https://medium.com/@ericdreichert/test-setup-and-teardown-in-rust-without-a-framework-ba32d97aa5ab

But IMO that should really be in a test framework …


#4

Wouldn’t this be considered a bug with hyper? Why isn’t the server destroyed/shut down (which would entail shutting down the background thread) when dummy_server is dropped at the end of the function? I’ve not used hyper, so maybe I’m missing something.

Why only when a panic occurs? I imagine you could have a hanging test case that succeeds but which doesn’t clean up background threads it launched.


#5

Looks like you’re not the only person having this problem with Hyper. https://github.com/hyperium/hyper/issues/338#issuecomment-264150856.


#6

I assume this is by design in Hyper. If you start a server like that then it “takes over” and will not shut down when the listening variable runs out of scope. How else would you start a server in your main function otherwise and keep it running?


#7

Listening.close() works perfectly fine for me - the problem is that the test execution never gets to my close() call because a panic has occurred.

What I need is a robust test framework that will always execute the teardown part of a test, no matter if the test function panics or misbehaves in any way.

I guess I have to invent such a test framework myself or do you know if something like that already exists?


#8

Here is an idea for a panic safe test framework. Structure one test case into 4 functions:

  1. a wrapper function that uses the test framework (maybe this could be avoided/simplified?)
  2. A set up function
  3. A function that has the test logic and makes assertions
  4. A teardown function to frees up resources (the started Hyper server in our case).
extern crate hyper;

use hyper::server::{Server, Request, Response, Listening};
use std::panic;

#[test]
fn test_test1() {
    test_run(set_up_test1, run_test1, tear_down_test1);
}

fn set_up_test1() -> Listening {
    let port = 9092;
    let dummy_server = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();
    return dummy_server;
}

fn run_test1() {
    let port = 9092;
    let _dummy_server2 = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();
}

fn tear_down_test1(mut listening: &mut Listening) {
    let _result = listening.close();
}

// This would be the test framework API function one can use for arbitrary
// tests.
fn test_run<S, R, T, X>(set_up: S, run: R, tear_down: T) -> ()
    where S: FnOnce() -> X + panic::UnwindSafe,
          R: FnOnce() -> () + panic::UnwindSafe,
          T: FnOnce(&mut X) -> () + panic::UnwindSafe
{
    // No panic catching in the setup function - if something goes wrong there
    // we are just out of luck.
    let mut x = set_up();
    let run_result = panic::catch_unwind(|| run());
    // Before we examine potential panics we teardown opened resources.
    tear_down(&mut x);
    run_result.unwrap();
}

This is no ideal, but the test case fails now as expected and does not hang, yay! The trick is to catch panics during the test run and only escalate them after invoking the teardown.

This is still less than ideal and I have no idea if the type stuff would even work if you have multiple resources that you need to clear up in the tear down phase.


#9

I don’t know of a test library that handles teardowns. I love your idea for using panic::catch_unwind.

It’s a bit verbose with the code split among so many functions. Maybe we can clean that up and keep things closer to a single test function.

The goals:

  1. Postpone the panic until after cleanup, or conversely, schedule cleanup on an unwind
  2. Let the test use the result that may have panicked for further assertions.

I think bluss’s scopeguard crate will work perfectly here.

#[macro_use]
extern crate scopeguard;
extern crate hyper;

use hyper::server::{Server, Response};

#[test]
fn test() {
    let server1 = Server::http("127.0.0.1:9092".to_string())
        .unwrap()
        .handle(|_, response| { response.send(b"hello").unwrap(); })
        .unwrap();
    defer!(server1.close());    // always runs when test returns or panics.

    let server2 = Server:http("127.0.0.1:9092".to_string())
        .unwrap()
        .handle(|_, response| { response.send(b"hello").unwrap(); })
        .unwrap();
    defer!(server2.close());    // panic should happen before this scopeguard is created. 
}

Not sure if you’ll have problems with the defer closures owning the server handles, though use of std::rc::Rc should compensate if that is the case.
Scopeguard shares the closure values per scopeguard’s tests.


#10

Thank you! scopegoard is definitely interesting and a quick nice hack to solve this problem, there is this example that works for me and fails as it should without hanging:

#[macro_use]
extern crate scopeguard;
extern crate hyper;

use hyper::server::{Server, Response, Request};

#[test]
fn test_port_overlap() {

    let port = 9092;
    // Test that starting the server fails if the port is already occupied.
    // Start a dummy server on port 9092 that just returns a hello.
    let mut dummy_server = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();
    defer!(dummy_server.close().unwrap());

    let _dummy_server2 = Server::http("127.0.0.1:".to_string() + &port.to_string())
        .unwrap()
        .handle(|_: Request, response: Response| { response.send(b"hello").unwrap(); })
        .unwrap();
}

I’m still wondering if there is a way for the cargo testrunner to kill all threads a test function has started as soon as it panics. Sounds like an obvious thing a testrunner should do.


#11

While rewriting my Hyper server for the new Tokio version all my problems just magically went away. The test case never hangs and the threads I’m spawing seem to terminate just fine when they run out of scope. No more .close() calls needed :slight_smile: