Capture test output in an object

[ethanabrooks]

I am writing an autograder in Rust. I would like to use Rust's inbuilt testing capabilities and integrate them with autograding infrastructure provided by Gradescope. Gradescope requires a JSON file with various annotations about the students' scores (for details: Autograder Specifications - Gradescope Autograder Documentation). I was hoping to generate this JSON file based on the results of cargo test, but this would require the ability to access those results programmaticaly. One solution is to scrape the result of cargo test from stdout, but I was hoping that there was a better way that didn't involve parsing strings.

I assume that Cargo builds some kind of programmatic representation of the test results before printing them to the terminal. Is it possible for me to access this programmatic representation somehow?

[daboross]

I think you can get this with a concoction of two options.

For cargo messages, --message-format json should give good stdout JSON output - this is for compile errors as well as reporting compile status of crates: cargo test - The Cargo Book

But the actual tests are compiled and run as separate binaries under cargo. To get them to output json, we need another option. Unfortunately it's only in an unstable form and under a flag on the nightly compiler. I have no idea where the docs for this are, but

cargo test -- -Z unstable-options --format json

should have the test binaries themselves output something like

{ "type": "test", "event": "started", "name": "src/builders.rs - builders::Dispatch::into_shared (line 354)" }
{ "type": "test", "name": "src/builders.rs - builders::Dispatch::filter (line 314)", "event": "ok" }
...

Combining this with --message-format json should give most output in json, then:

cargo test --message-format json -- -Z unstable-options --format json

It'll still output some status things to stderr, but as far as I can tell this outputs all compile errors well as test results as JSON. I... don't know where documentation is for this, but I don't think it changes often - and if you fix your test nightly version it should be doable!

I also found the cargo_metadata crate which may or may not be helpful in parsing this.

Let me know how this goes!

[ethanabrooks]

This is very useful! cargo_metadata seemed primarily intended to pull out build information, not to actually run a command like cargo test. This is what I came up with

use std::process::Command;

pub fn get_test_output(path: String) -> String {
    //cargo test --manifest-path="../../Cargo.toml"  -- -Z unstable-options --format json -q
    let stdout = Command::new("cargo")
        .arg("test")
        .arg(format!("--manifest-path={}", path))
        .arg("--")
        .arg("-Z")
        .arg("unstable-options")
        .arg("--format")
        .arg("json")
        .output()
        .expect("Failed to capture output")
        .stdout;
    String::from_utf8(stdout).expect("Failed to convert stdout to string.")
}