Slightly scary line about testing binary crate?

1

If our project is a binary crate that only contains a src/main.rs file and doesn’t have a src/lib.rs file, we can’t create integration tests in the tests directory and bring functions defined in the src/main.rs file into scope with a use statement. Only library crates expose functions that other crates can use; binary crates are meant to be run on their own.

That's fine. But then,

If the important functionality works, the small amount of code in the src/main.rs file will work as well, and that small amount of code doesn’t need to be tested.

I'm unsure why wouldn't one test the complete thing? Just the last line sounds scary, but I'm unsure I understand it. I'd still test some one or two lines left.

2

In a typed language, it really doesn't seem to be the best use of developer time to write tests for a main function that's basically

use my_app::real_main;
fn main() {
    real_main();
}

plus maybe a few lines of error reporting if needed.

3 Likes
3

My understanding was, that "integration tests" just do not work for binary (application) crates. So this is the only possible solution.

4

Heya, it usually boils down to one thing:

If we had unlimited time, we would do everything.

Unfortunately we don't.

There's a number of perspectives to consider:

  1. Effort (maintenance / mental):

    If lib.rs and the supporting 1000 lines are tested, and the 2 lines in main.rs, it usually gives enough assurance that things work, and if something breaks, we can easily reason over the 2 lines in main.rs, and the tests will notify us for the 1000 other lines.

  2. These tests do exist!

    In the cases where main.rs code is actually large, many people have been on journey of writing tests that run actual binaries, and asserting output.

    One example is clap's ui tests, where the test cases are actually defined as toml, and ui.rs uses trycmd to assert the output of the commands. Testing binaries got big enough to warrant writing a separate runner to make it easier

  3. Some things are just harder to do with binary tests, and not everyone would spend time on that.

    • With binary tests, it can be harder to collect (code) coverage if the test runner doesn't support running the binary wrapped in the code coverage tool.
    • If the binary needs a graphics card / audio to run, then setting up CI with a visual frame buffer / audio support with the right settings can be a nightmare. I really became frustrated about that and nearly quit many times. (40 hours to get one command working is.. pain)

Also, hobbies should be fun, not always a grind, or perfection :slightly_smiling_face:.

5

@jdahlstrom I take the perspective to an extent. But the part:

isn't sound to me. Testing one line isn't time consuming, and if you import the wrong function by mistake, it is a bug.

Conversely, would one not test all one line functions?

@StefanSalewski Yes, that's correct. But the only possible solution doesn't mean one should advice not to test that one line. This seems to me a debatable advice to give. Especially when they start with (bold is mine):

In his 1972 essay “The Humble Programmer,” Edsger W. Dijkstra said that “program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” That doesn’t mean we shouldn’t try to test as much as we can!

@azriel91 I take what you say about the effort it might take to implement, but not the advice of not testing two lines, especially those are critical. Thanks for the links and explanations in 3..

As for this I'm unsure what it means? If you plan a serious application, you'd simply aspire to test as much as you can, or at least some of us?

6

you sound like a younger me :slightly_smiling_face:, here's my "ticket" showing I was passionate about as much coverage as possible:

  • fn_graph -- 99% coverage (5915 lines, 47 missed)
  • peace -- 88% coverage (41390 lines, 5172 missed).

with that precursor, here are some situations where I don't normally get 100% coverage, and their reasons:

  1. Server binaries

    i.e. a binary where there's an infinite loop, with a socket listening for connections and serving them.

    It is certainly a demonstration of diligence and high quality to add a shutdown mechanism (e.g. interrupt signal with graceful shutdown), in which a test can invoke and wait on the binary in one thread, and send in test data + send the shutdown signal in another thread.

    However, if the project's most important value is still being discovered (e.g. dot_ix, I'm still figuring out how to draw good diagrams), then I would devote what time I have for side projects to the drawing of the diagram, than the testing of the socket listener.

  2. Frequently churning code

    This first part isn't limited to binaries: if I have some code that is changing frequently, because I:

    • don't know what "correct" is (e.g. still figuring out API function signatures), and
    • I write 10 tests for that code

    whenever the code changes, I have to update those tests (add/remove/update). The changing of those tests is effort, which may mean I spend 10 hours making 10 changes because I have to keep the tests up to date, instead of 5 hours for 10 changes.

    So in a known unstable state, I tend to forego tests while I discover what things should be, and when a good design reveals itself and I'm ready to consider it stable(ish), then I'll add tests to catch any unexpected breakages I cause.

    For binaries specifically, this could be the CLI arguments, or output, and I might only want to write assertions when the argument format / output is stable-ish.

There's probably more :grinning_face:

edit: forgot to address this bit

Some people might want to code a game as a "serious" hobby project, and may not necessarily choose to write tests. And that's okay right? There isn't any obligation how to do a personal project.

2 Likes
7

to OP. It makes more sense not split out from the full paragraph.
It would be much better served by saying such code will be covered by end to end testing so does not need a duplication of effort.
(I personally am happy to have the executable run by what should be integration tests when project is small enough.)

tests are still important to reduce logic bugs

The trouble with many writings is they miss the point that testing is there to find bugs.

Testing is an allocation of time. The goal can be to maximise the number of bugs you can find. Until AI takes over, it is likely the regression testing of unit and integration automation is the best rule for the majority of projects. Finding bugs early is the approach many view as most worthwhile. (I'm not a fan of test driven development though.)

1 Like
8

Integration tests are not unit tests.

The public API of a binary is the binary, not library interfaces.

You can completely test binaries in integration tests — by running them.

7 Likes
9

So one makes it into a library and runs the binary from an integration test.

  • Is this interpretation correct?

Imho this is much solid than not testing it.

I am aware that IT are not UTs, and dont have same purpose.

But there in the quote I understood the book talks in general. (And both cases were explained already. But no mention about this usage for the binary.)

  • Or are you saying they didn't mean that in the quoted text?
10

What you're complaining about in the book should probably be improved. Please create an issue for it and ask that it be softened, and perhaps that an integration test could be used. For example:

If the important functionality works, the small amount of code in the src/main.rs file may not need testing, or could be tested with an integration test.

11

I don't use GitHub. Apologies for the complain —if it reads like that is simply some frustration, not anything else. Plus being unsure about communication.

12

Ok.

I created a very small PR for this.

What bugs me sometimes is when effort is put into a discussion and nothing comes out of it. I feel better now. :^)

2 Likes
13

or can be tested by executing the binary from the integration test.

Is my understanding correct, that this would mean that we copy main.rs or the other binary sources from the bins folder into the tests directory, where they are then run by "cargo test"? That might work, when main.rs needs no parameters, and is not an app like an editor that needs user input, and does not terminate on its own.

Or do you suggest that we create in the tests folder some form of build command, that compiles main.rs, and executes it as a new process, with optional parameters?

14

You can use the env!("CARGO_BIN_EXE_<name>") listed on this page to directly run the compiled binary using std::process::Command

4 Likes
15
  • Do we need to have a lib.rs as well? I suspect from here

That we don't. But maybe is a misinterpretation.

Either way, I think a more explicit example would look like:

main.rs 
use my_printer; // crate name

fn main() {
    my_printer::print_hi();
}
`tests/the_bin.rs`

Adapting from Command in std::process - Rust, which shows how to pass flags as well:

let bin = env!("CARGO_BIN_EXE_my_printer"); // our binary

let output = Command::new(bin)
        .output()
        .expect("failed to execute process")

let hello = output.stdout;

In my view, a refined version of this is what the book needs, given who the readers are (beginners.)

16

OK, I have added your suggestion to Test organization - Rust for C-Programmers for now so that I will not forget it. I have still to test it to ensure that it really works, and I am still a bit unsure if the required space is justified. But it is just an addition, so I can remove it later easily.

17

Maybe someone here can review it, I dont have the knowledge.

I didn't test that, but before adding to any book, it should be imho.

It does look good though, I would only add a link to the command page in the standard library.

1 Like
18

Another related link (sp last paragraph in section) Cargo Targets - The Cargo Book

Binary targets are automatically built if there is an integration test. This allows an integration test to execute the binary to exercise and test its behavior. The CARGO_BIN_EXE_ environment variable is set when the integration test is built so that it can use the env macro to locate the executable

19

Thanks for that information. Actually, yesterday I already wondered what might happen if we try to run our binary in our integration test, but the binary does not exists. As a non native speaker, I am not absolutely sure how I can interpret the sentence "Binary targets are automatically built if there is an integration test." Is this like "... if there is any integration test.", so that if there exists any integration tests in the tests folder, it is guaranteed that the binaries are build, and we can run them? Note I use plural, because additional to main.rs there can exist additional binaries in the bins folder that we might test as well.

20

If it there isn't a binary the environment variable will be "" which is the default value. will not be defined and:

If the environment variable is not defined, then a compilation error will be emitted.

So it fails.