Async tests sometimes fails

I implemented some tests for TCP client that uses tokio. When I run cargo test tests sometimes OK, sometimes some of them failed (the issue is related only to async test with client.connect() call).

This is my tests:

#[cfg(test)]
mod tests {
    use std::io::{Read, Write};
    use tokio::io::{AsyncReadExt, AsyncWriteExt};
    use tokio::net::TcpListener;
    use crate::Client;
    use crate::client::types::ClientFlags;
    use crate::network::session::types::{ActionFlags, StateFlags};

    const HOST: &str = "127.0.0.1";
    const PORT: u16 = 1234;
    const WRONG_HOST: &str = "1.2.3.4";
    const PACKET: [u8; 8] = [1, 2, 3, 4, 5, 6, 7, 8];

    #[tokio::test]
    async fn test_client_create() {
        let client = Client::new();

        let reader = &mut *client.reader.lock().await;
        assert!(reader.is_none());

        let writer = &mut *client.writer.lock().await;
        assert!(writer.is_none());

        let warden_crypt = &mut *client.warden_crypt.lock().await;
        assert!(warden_crypt.is_none());

        let client_flags = &mut *client.flags.lock().await;
        assert_eq!(&mut ClientFlags::NONE, client_flags);

        let input_queue = &mut *client.input_queue.lock().await;
        assert!(input_queue.is_empty());

        let output_queue = &mut *client.output_queue.lock().await;
        assert!(output_queue.is_empty());

        let data_storage = &mut *client.data_storage.lock().await;
        assert!(data_storage.players_map.is_empty());

        let session = &mut *client.session.lock().await;
        assert!(session.session_key.is_none());
        assert!(session.me.is_none());
        assert!(session.warden_module_info.is_none());
        assert!(session.config.is_none());
        assert!(session.follow_target.is_none());
        assert!(session.server_id.is_none());
        assert!(session.party.is_empty());
        assert_eq!(ActionFlags::NONE, session.action_flags);
        assert_eq!(StateFlags::NONE, session.state_flags);

    }

    #[tokio::test]
    async fn test_client_connect() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            client.connect(HOST, PORT).await.ok();

            let reader = &mut *client.reader.lock().await;
            assert!(reader.is_some());

            let writer = &mut *client.writer.lock().await;
            assert!(writer.is_some());
        }
    }

    #[tokio::test]
    #[should_panic]
    async fn test_client_connect_with_wrong_data() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            client.connect(WRONG_HOST, PORT).await.unwrap();
        }
    }

    #[tokio::test]
    async fn test_client_read_incoming_data() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            client.connect(HOST, PORT).await.ok();

            if let Some((mut stream, _)) = listener.accept().await.ok() {
                stream.write(&PACKET).await.unwrap();
                client.handle_read().await;

                let packet = client.input_queue.lock().await.pop_front().unwrap();
                assert_eq!(PACKET.to_vec(), packet[0]);
            }
        }
    }

    #[tokio::test]
    async fn test_client_write_outcoming_data() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            client.connect(HOST, PORT).await.unwrap();

            if let Some((mut stream, _)) = listener.accept().await.ok() {
                let mut buffer = Vec::new();

                client.output_queue.lock().await.push_back(PACKET.to_vec());
                client.handle_write().await;
                stream.read(&mut buffer).await.unwrap();

                assert_eq!(PACKET.to_vec(), buffer);
            }
        }
    }
}

could somebody help to find what is wrong ? Next tests sometimes fail: test_client_connect_with_wrong_data, test_client_write_outcoming_data and test_client_read_incoming_data

Just in case my code is implemented here: idewave-cli/mod.rs at dev · idewave/idewave-cli · GitHub

(Just a guess of problem cause) Tests run in parallel.

1 Like

What error do you get when they fail?

With the tests running in parallel (in separate threads), each thread might try to bind to HOST, PORT concurrently, resulting in a race condition. In that case, if let Some(listener) will silently discard the error and the body of some tests will not get a chance to run.

2 Likes

most common error with test_client_connect_with_wrong_data is next:

running 5 tests
test client::tests::test_client_create ... ok
test client::tests::test_client_connect_with_wrong_data - should panic ... FAILED
test client::tests::test_client_write_outcoming_data ... ok
test client::tests::test_client_read_incoming_data ... ok
test client::tests::test_client_connect ... ok

failures:

---- client::tests::test_client_connect_with_wrong_data stdout ----
note: test did not panic as expected

when I run then cargo test --bin idewave-cli several times, I got error from another test test_client_write_outcoming_data too:

  left: `[1, 2, 3, 4, 5, 6, 7, 8]`,
 right: `[]`', src\client\mod.rs:435:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    client::tests::test_client_connect_with_wrong_data
    client::tests::test_client_write_outcoming_data

test result: FAILED. 3 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

error: test failed, to rerun pass '--bin idewave-cli'

also sometimes I got this error:

running 5 tests
test client::tests::test_client_create ... ok
test client::tests::test_client_connect ... ok
test client::tests::test_client_write_outcoming_data ... ok
test client::tests::test_client_connect_with_wrong_data - should panic ... FAILED
test client::tests::test_client_read_incoming_data ... FAILED

failures:

---- client::tests::test_client_connect_with_wrong_data stdout ----
note: test did not panic as expected
---- client::tests::test_client_read_incoming_data stdout ----
Connected to 127.0.0.1:1234
thread 'client::tests::test_client_read_incoming_data' panicked at 'called `Option::unwrap()` on a `None` value', src\client\mod.rs:416:74
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    client::tests::test_client_connect_with_wrong_data
    client::tests::test_client_read_incoming_data

test result: FAILED. 3 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

could you clarify why this can be an issue ? Btw, how to be sure of this ? Is it because tokio use rt-multi-thread in dependencies ?

just found, tests are runned always in different order. In case test_client_connect_with_wrong_data is last in the order, all tests are OK. Tried multiple times and it was reproduced multiple times. Not sure if this can be a reason.

Tests should always bind their TcpListener to port zero. When you do this, the operating system chooses a random unused port for you.

3 Likes

when I set port to "0" I got next error:

running 5 tests
test client::tests::test_client_create ... ok                                                                                                                                                                             
test client::tests::test_client_connect_with_wrong_data - should panic ... ok                                                                                                                                             
test client::tests::test_client_read_incoming_data ... FAILED                                                                                                                                                             
test client::tests::test_client_write_outcoming_data ... FAILED                                                                                                                                                           
test client::tests::test_client_connect ... FAILED                                                                                                                                                                        
                                                                                                                                                                                                                          
failures:                                                                                                                                                                                                                 
                                                                                                                                                                                                                          
---- client::tests::test_client_read_incoming_data stdout ----                                                                                                                                                            
thread 'client::tests::test_client_read_incoming_data' panicked at 'Cannot connect: Os { code: 10049, kind: AddrNotAvailable, message: "The requested address is not valid in its context." }', src\client\mod.rs:112:17  
                                                                                                                                                                                                                          
---- client::tests::test_client_write_outcoming_data stdout ----                                                                                                                                                          
thread 'client::tests::test_client_write_outcoming_data' panicked at 'Cannot connect: Os { code: 10049, kind: AddrNotAvailable, message: "The requested address is not valid in its context." }', src\client\mod.rs:112:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- client::tests::test_client_connect stdout ----
thread 'client::tests::test_client_connect' panicked at 'Cannot connect: Os { code: 10049, kind: AddrNotAvailable, message: "The requested address is not valid in its context." }', src\client\mod.rs:112:17


failures:
    client::tests::test_client_connect
    client::tests::test_client_read_incoming_data
    client::tests::test_client_write_outcoming_data

test result: FAILED. 2 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

also I tried to set 0 port for listener and 1234 port for client.connect

if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, 0)).await.ok() {
    client.connect(HOST, 1234).await.unwrap();
}

and got error:

---- client::tests::test_client_write_outcoming_data stdout ----
thread 'client::tests::test_client_write_outcoming_data' panicked at 'Cannot connect: Os { code: 10061, kind: ConnectionRefused, message: "No connection could be made because the target machine actively refused it." }', src\client\m
od.rs:112:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- client::tests::test_client_connect stdout ----
thread 'client::tests::test_client_connect' panicked at 'Cannot connect: Os { code: 10061, kind: ConnectionRefused, message: "No connection could be made because the target machine actively refused it." }', src\client\mod.rs:112:17 

Well, if you have the OS pick a random port, then you can't just keep using the port 1234. The test would only succeed if the random choice happened to be 1234. You can access the port it chose using TcpListener::local_addr.

1 Like

well, now I do this:

#[tokio::test]
    async fn test_client_write_outcoming_data() {
        let mut client = Client::new();
        // PORT == 0 here
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            // connect to random port
            let local_addr = listener.local_addr().unwrap();
            client.connect(HOST, local_addr.port()).await.ok();

            client.output_queue.lock().await.push_back(PACKET.to_vec());

            if let Some((mut stream, _)) = listener.accept().await.ok() {
                let mut buffer = Vec::new();

                // here writer.write(&packet).await; send not-empty packet
                client.handle_write().await;
                // but stream return empty buffer here
                stream.read(&mut buffer).await.unwrap();
                assert_eq!(PACKET.to_vec(), buffer);
            }
        }
    }

and seems like it helped with random test fail, but now stream not work. It not read or write. So I debugged and sure that inside client.handle_write().await; I have packet, but next line stream.read(&mut buffer).await.unwrap(); not work. It just return empty buffer. Could you help on this ?

I fixed one test by adding read_to_end:

#[tokio::test]
    async fn test_client_write_outcoming_data() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            let local_addr = listener.local_addr().unwrap();
            client.connect(HOST, local_addr.port()).await.ok();
            client.output_queue.lock().await.push_back(PACKET.to_vec());

            if let Some((mut stream, _)) = listener.accept().await.ok() {
                let buffer_size = PACKET.to_vec().len();
                let mut buffer = Vec::with_capacity(buffer_size);

                client.handle_write().await;
                stream.take(buffer_size as u64).read_to_end(&mut buffer).await.unwrap();

                assert_eq!(PACKET.to_vec(), buffer);
            }
        }
    }

but for now not sure how to use same approach with write. Tried to add stream.flush().await.unwrap(); but this not helped.

fixed another one by adding loop:

#[tokio::test]
    async fn test_client_read_incoming_data() {
        let mut client = Client::new();
        if let Some(listener) = TcpListener::bind(format!("{}:{}", HOST, PORT)).await.ok() {
            let local_addr = listener.local_addr().unwrap();
            client.connect(HOST, local_addr.port()).await.ok();

            if let Some((mut stream, _)) = listener.accept().await.ok() {
                stream.write(&PACKET).await.unwrap();
                stream.flush().await.unwrap();
                client.handle_read().await;

                loop {
                    if let Some(packet) = client.input_queue.lock().await.pop_front() {
                        assert_eq!(PACKET.to_vec(), packet[0]);
                        break;
                    }
                }
            }
        }
    }

thanks @alice for detailed explanation ! And thanks to all who help !

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.