Ideas for writing portable tests executing external programs?


#1

I’m currently working on getting more CI working for bigbro, and am struggling a bit to get the doctests to work. Since bigbro is all about running external commands, I need some external commands to actually run! I have a set of commands that I’ve been using on linux, and most of them work (with tweaking) for Mac, but windows is another story.

Any suggestions for useful portable commands? Or for a nice way to organize the doctests to use non-portable commands? I’ve already switched to using std::env::temp_dir() to increase portability, but the trouble is that the documentation gets more challenging to understand as I start adding more checks in there, so I don’t see a match to do different things on different platforms a great option.

As a more concrete question: sha1sum seems to be portable, but I need to find a file to take the checksum of. Any suggestions for an easy way using the rust standard library to identify some file, any file? Currently I’m just using /usr/bin/python, which obviously won’t work on windows.

For the curious, you could see how (poorly?) my testing is going:

https://ci.appveyor.com/project/droundy/bigbro
https://travis-ci.org/droundy/bigbro


#2

Perhaps you could use your own commands built with Rust?

You could add a project with commands to the workspace and declare it as a dev dependency, which would avoid adding the commands to crates.io and release version of your project.


#3

Would that be possible to do this without creating a new crate, e.g. by creating the commands as tests or something? Any idea how to get them into the PATH when running doctests?


#4

IIRC binaries of the crate are built before tests. But to separate test dummies from real project binaries it may be best to use another crate anyway. With workspaces it’s easy.


#5

I’m trying to put test binaries into a separate crate with a workspace, but am running into trouble where they are not getting built. My test crate (which has a src/bin/echo.rs) is listed in the main Cargo.toml as:

[dev-dependencies]

test-utils = { path = "test-utils" }

If I cd into test-utils and run cargo build then the echo executable is built and put in the root target/debug/echo, so the workspace is working properly, but when I run either cargo build or cargo test from the root, the test-utils crate is not built, and echo is not Any suggestions for debugging this?


#6

Try adding extern crate test_utils; to tests/crates that need it.


#7

Even though test-utils doesn’t have a library?!


#8

If I understand that correctly it should work for integration tests (I’m assuming that’s tests/foo_test.rs):



#9

Those links are helpful, but don’t address the separate crate issue (with a workspace), which is what I’m running into. Probably it’s not so super easy, and I should just generate excess binaries in my single crate.


#10

Also, any suggestions for setting the PATH when running tests? Specifically in doc-tests?


#11

Maybe @alexcrichton will be able to help how to test with binaries from a sub-crate in a workspace.


#12

Ah unfortunately I don’t think there’s a way to do that right now, have binaries from another member of the workspace available when you test the main crate. This may be possible with cargo test -p foo -p bar to “test both crates” but actually just pick up binaries from one crate.