Testing Long-Running CLI app…

Caveat: I’m in the early-learning stages of my Rust journey. Have mercy :D.

I’m in desperate need of some guidance. I’m struggling to use my pattern of test-driven development with Rust. I’ve got a little pet project CLI-app that I’m building - and while I am getting smarter/better at the implementation bits (including project layout) to make things more testable, I’m stumped on one aspect. How to test something that is scheduled to run in the future, and spins out a thread, outputting something to stdout/stderr as its functionality. So I have a few “unit” tests, and some “functional” tests that execute the actual CLI’s output. But not sure how to approach the bits that run forever. And maybe I’m chasing ghosts here, and should just move on and implement things and test them manually. Just can’t help think there’s some good design practices, and tools that I’m in search of.

Any help would be greatly appreciated. Or pointers to projects that might already be doing these things so I can learn on those shoulders.

TIA,
Kit

What you are looking for is an integration test, if I am not mistaken. Now the thing with Rust's own built in testing system is that if you have tests which do not have a well defined upper bound on when it will finish, it becomes a pain to run them.
The way I see it, there seems to be two general ways to approach this problem:

  • Write the tests as integration tests using Rust's testing system. Write wrapper scripts to choose to not run the long running tests when you are running unit tests.
  • Write the tests as examples. Then compare the output with an expected golden using a separate script.

Here’s an example of the “functional” test pattern:


#[test]
fn test_help_command() -> Result<(), Box<dyn std::error::Error>> {
    let mut cmd = Command::cargo_bin("goa")?;
    cmd.arg("help");
    cmd.assert().stdout(predicates::str::contains(
        "Help!!!",
    ));
    Ok(())
}

Is this what you’re referring to as integration @RedDocMD ?

This works great for sub-commands that return instantly, and I’ve also got a way to look at the stderr and status too. But, the trick I need is - maybe to sigint to stop the command and then evaluate stdout/stderr/status code…

You don't. At least, not directly.

Instead, you probably want to split it up into smaller pieces that can be tested independently

  • Run an arbitrary function at an arbitrary point in time (requires mocking out "time")
  • The business logic for your bunction being tested (use Dependency Injection so instead of printing to stdout your function calls a method on an object which your tests can hook into)
  • The logic for scheduling tasks and saying when a task should be executed next (requires separating your task execution code from the code managing scheduling)

Depending on the importance of this whole system (production app making millions of dollars, toy project, etc.) you can then look into a full integration test which runs your CLI app in a special environment and makes sure the right things happen at the right time in real time (e.g. using sleeps and the assert_cmd crate). This is overkill for a toy project because it means your test may take many minutes, so normally I'd just test it manually.

1 Like

Yes, this is sort of the dilemma…might take more effort to implement automated test than to implement the functionality. Not abnormal really, just have to weigh the efforts.

I’m definitely working to isolated the bits that get scheduled away from the scheduler itself in order to get them tested.

WRT the functional tests on the app itself, I can definitely have them only run when desired…like on a release/PR/MR process.

It depends on how you designed your code when starting out, but I don't think this should add much extra effort.

You will normally want to separate the logic for scheduling tasks, executing tasks, and the tasks themselves, anyway, because that makes your code easier to reason about/debug and lets you reuse things.

If I were designing such an app, I might do something like this...

First we create an internal Task type which will encapsulate the thing being executed by our scheduler

use std::time::{Duration, Instant};

struct Task {
    next_run: Instant,
    func: Box<dyn FnMut() -> NextRun>,
}

pub enum NextRun {
    After(Duration),
    Halt,
}

The comes the Scheduler - something which contains a list of Tasks. You run it to completion by continually popping tasks from the list, sleeping until they need to be run, then invoking the func callback and using the returned value to determine whether the task should be re-scheduled or not.

[#[derive(Default)]
struct Scheduler {
    tasks: Vec<Task>,
}

impl Scheduler {
    pub fn schedule(&mut self, delay: Duration, func: Box<dyn FnMut() -> NextRun>) {
        let next_run = Instant::now() + delay;
        self.tasks.push(Task { next_run, func });
        self.tasks.sort_by_key(|t| t.next_run);
    }

    pub fn start(&mut self) {
        while let Some(task) = self.next_task() {
            self.execute_task(task);
        }
    }

    fn next_task(&mut self) -> Option<Task> {
        if self.tasks.is_empty() {
            None
        } else {
            Some(self.tasks.remove(0))
        }
    }

    fn execute_task(&mut self, task: Task) {
        let Task { mut func, next_run } = task;

        let time_to_sleep = next_run.saturating_duration_since(Instant::now());
        std::thread::sleep(time_to_sleep);

        match func() {
            NextRun::After(delay) => {
                self.schedule(delay, func);
            }
            NextRun::Halt => {}
        }
    }
}

The main() function then configures the Scheduler and calls its start() method.

fn main() {
    let mut scheduler = Scheduler::default();
    scheduler.schedule(Duration::from_millis(50), create_task_func());

    scheduler.start();
}

For demonstration purposes I'm using a helper to create the actual task function being executed, but that's mainly because I want the scheduler to stop after a while so the playground doesn't abort the process.

/// A dummy function which creates a task function that prints messages
/// periodically and stops after 1 second.
fn create_task_func() -> Box<dyn FnMut() -> NextRun> {
    let created = Instant::now();

    Box::new(move || {
        let time_since_created = Instant::now() - created;
        println!("Calling after {:?}", time_since_created);

        if time_since_created.as_secs() >= 1 {
            // Make sure we eventually stop so the playground doesn't abort the
            // process
            NextRun::Halt
        } else {
            NextRun::After(Duration::from_millis(50))
        }
    })
}

(playground)

This might sound like a lot of code but chances are you have all the same code and logic already, it's just that my version splits it out into separate types and functions that can be tested individually separately.

1 Like