How do you test a library imported by another program?

I'm working on a crate that gets compiled as a C dynamic library and imported by an another binary (Neovim in this case).

I'd like to leverage cargo test to test it, but I'm unsure what the less inconvenient option might be.

My current solution involves a 2-step process (code is here):

  1. compile the code snippet I'd like to test, producing a .so file. The code looks something like:
use nvim_oxi::lua::lua_State;
    
#[no_mangle]
extern "C" fn luaopen_liboxi_tests(_state: *mut lua_State) -> std::os::raw::c_int {
    let result = std::panic::catch_unwind(|| {
        // Code I want to test.
    });

    std::process::exit(match result {
        Ok(_) => 0,
    
        Err(err) => {
            eprintln!("{err:?}");
            1
        },
    })
}
  1. write a regular test that starts Neovim with the appropriate cli flags and asserts that stderr is empty:
#[test]
fn () {
    let out = std::process::Command::new("nvim")
        .args(["-u", "NONE", "--headless"])
        .args(["-c", "set rtp+=/Users/noib3/Dropbox/projects/nvim-oxi"])
        .args(["-c", "lua require(\"liboxi_tests\")"])
        .args(["+quit"])
        .output()
        .expect("Couldn't find `nvim` binary in $PATH!");
    
    let stderr = String::from_utf8_lossy(&out.stderr);
    
    assert_eq!(stderr, String::new());
}

This is obviously not ideal for a variety of reasons:

a) very verbose, everything apart from the the // Code I want to test. line is noise. Could be avoided by writing a proc macro that extends #[test];

b) stderr output are nested in the event of an unsuccessful test, for example

 ---- api::buffer::get_changedtick stdout ----
thread 'api::buffer::get_changedtick' panicked at 'assertion failed: `(left == right)`
  left: `"thread '<unnamed>' panicked at 'assertion failed: buf.get_changedtick().is_er
  r()', oxi-tests/src/api/buffer.rs:43:9\nnote: run with `RUST_BACKTRACE=1` environment v
  ariable to display a backtrace\nAny { .. }\n"`,
  right: `""`', oxi-tests/src/api/buffer.rs:32:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

notice the two thread <x> panicked at <y>. I'd like to only see the inner one;

c) it's a 2 step process, I have to do a cargo build before every cargo test *. Could be avoided with a simple shell alias;

d) since every test has to compiled into a separate crate, if I have 100 integration tests I'll have to wait for 100 cargo builds before getting all the results. This is probably the worst part and I don't think there's a lot one can do to avoid it :confused:

I'd appreciate some input on the various points. Is there a better way that I'm not seeing?

Original Reddit post.

Well, a small macro may help, like:

#[cfg(test)]
macro_rules! test_stderr {
    ($($args:expr),*) => {{
        let out = ::std::process::Command::new("nvim")
            .args(["-u", "NONE", "--headless"])
            .args(["-c", "set rtp+=/Users/noib3/Dropbox/projects/nvim-oxi"])
            $(
                .args($args)
            )*
            .args(["+quit"])
            .output()
            .expect("Couldn't find `nvim` binary in $PATH!");

        String::from_utf8_lossy(&out.stderr).into_owned()
    }};
}

Due to the assert_eq! on failed tests, "assertion failed: (left == right)" is the noise for you. But the insta snapshot-testing tool may help, via cargo insta test --review :

#[test]
fn get_changedtick_with_macro() {
    let stderr = test_stderr!(["-c", "lua require(\"liboxi_tests\")"]);

    insta::assert_display_snapshot!(stderr, @r###"
    thread '<unnamed>' panicked at 'attach failed: NvimError("unexpected key: on_reload\xab<+\x0c\x7f")', src/api/buffer.rs:16:48
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    Any { .. }
    "###);
} 

Or as an alternative, trybuild is designed to test the stderr/stdout diagnostics.

A test.sh script file helps, and you can run sh test.sh to reduce keystrokes.
BTW before running .args(["-c", "set rtp+=/Users/noib3/Dropbox/projects/nvim-oxi"]), you have to mkdir lua under that path, because lua require() searches for modules in lua/ folders in your 'runtimepath'.

Even if you've got a .so file without automatically testing it, you actually need to test it by hand to check whether it works.
What's more, cargo-nextest can speed up the testing.

For showing the snapshot to review:

#[cfg(test)]
macro_rules! test_stderr {
    ($($args:expr),*) => {{
        let out = ::std::process::Command::new("nvim")
            .args(["-u", "NONE", "--headless"])
            .args(["-c", "set rtp+=/rust/tmp/nvim-oxi/oxi-tests/target/debug"])
            $(
                .args(["-c", $args])
            )*
            .args(["+quit"])
            .output()
            .expect("Couldn't find `nvim` binary in $PATH!");

        String::from_utf8_lossy(&out.stderr).into_owned()
    }};
}

#[test]
fn get_changedtick_with_macro() {
    insta::assert_display_snapshot!(test_stderr!("lua require(\"liboxi_tests\")"), @r###"
    thread '<unnamed>' panicked at 'attach failed: NvimError("unexpected key: on_reload{eK\\\x7f")', src/api/buffer.rs:16:48
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    Any { .. }
    "###);
} 
Reviewing [1/2]:
Package: oxi-tests (0.1.0)
Snapshot: get_changedtick_with_macro
Source: src/lib.rs:43
-old snapshot
+new results
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test_stderr!("lua require(\"liboxi_tests\")")
────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    0       │-thread '<unnamed>' panicked at 'attach failed: NvimError("unexpected key: on_reload{eK\\\x7f")', src/api/buffer.rs:16:48
          0 │+thread '<unnamed>' panicked at 'attach failed: NvimError("unexpected key: on_reload{\xf5\x89f\x7f")', src/api/buffer.rs:16:48
    1     1 │ note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    2       │-Any { .. }
          2 │+Any { .. }
────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  a accept   keep the new snapshot
  r reject   keep the old snapshot
  s skip     keep both for now
1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.