Piping the animation of an external command back into stdout

Hi,

I'm currently writing a wrapper around some nix commands. Some of these can run for a while, and their only output for large stretches of time is a sort of animated status, looking something like this:

image

and updating in real time. Depending on the task and machine, this can take 30min+, making it really nice to have something on screen indicating progress, esp. since it's sometimes not clear how long things will take before running the command.

Unfortunately, and for some godforsaken reason I will never understand, nix outputs this to stderr. I say "unfortunately" because I need to do some filtering on stderr.

Something like this:

    pub(crate) fn run_filtered(&self) -> Result<()> {
        let (command, args) = self.get_final_args();
        let mut child = Command::new(command)
            .args(args)
            .stdout(Stdio::inherit())
            .stderr(Stdio::inherit())
            .spawn()?;

        Ok(())
    }

works perfectly for the first requirement (output everything, incl. the "animated" status), but of course does not allow me to filter anything.

On the other hand, something like

    pub(crate) async fn run_filtered(&self) -> Result<()> {
        let (command, args) = self.get_final_args();
        let mut child = Command::new(command)
            .args(args)
            .stdout(Stdio::inherit())
            .stderr(Stdio::piped())
            .spawn()?;

        const NEWLINE: u8 = 0xa;
        let mut lines = BufReader::new(child.stderr.take().unwrap());

        loop {
            let mut line = Vec::new();
            let len = lines.read_until(NEWLINE, &mut line).await?;
            let line = String::from_utf8_lossy(&line);

            if len == 0 {
                break;
            }

            // ...do the filtering...

           println!("{}", line); // same for print!()
        }

        Ok(())
    }

does the filtering nicely, but I completely loose the "animated" parts (screen just stays blank). I assume this is due to reading until the newline symbol, when the animation is probably just outputting a \r to update, but replacing the 0xa with 0xd or b'\r' just stops output altogether.

I'm still relatively new to rust, and feel like I've hit my current limit here. If someone could point me in the right direction, I'd really appreciate it :slightly_smiling_face:

There are other terminal codes than \r to move to the beginning of the line. You may need to capture and examine the output. Even then there might not be a great solution depending on the details. For example, there may not be a common delimiter that works for both the status redraws and the other non-status outputs. There isn't a general solution for every redrawing command output.

If there's a way to get the commands to print out a line per update instead redrawing, say, you may be better off.

Unfortunately, and for some godforsaken reason I will never understand, nix outputs this to stderr.

It does this so that you can use tools like grep and less to read its actual output, without having to disentangle it from the auxilliary status information. I'd even argue that this is a reasonable and correct use of stderr.

I assume this is due to reading until the newline symbol

I would second-guess this assumption. Check if your program is reading anything at all. It might not be - nix might not be writing anything for it to read.

It's fairly common for programs that do fancy terminal tricks to inspect the handle those outputs will go to (in this case, stderr). I would expect nix to print status animations only if its stderr is a TTY, and to suppress those animations or replace them with non-animated output if stderr is a non-TTY file handle. A non-TTY handle would suggest an output device (or a file, or a socket) that may not support terminal control codes in the first place, and without terminal code support, those animations become nonsensical.

You can check nix's behaviour pretty easily by running nix …options… |& cat - that will redirect both stderr and stdout to a program (cat) before sending them to your terminal, and I would be unsurprised if that turns off nix's animations. A pipe is not a TTY, and both your shell and your Rust program would use a pipe for the case you're interested in.

Your options for how to proceed, if I'm right about any of this, would depend on what you're trying to do with nix's output.

First, if you're trying to process its status output to perform some mechanical process on it, such as tracking time remaining, you might want to talk to the nix authors or read the documentation for alternatives. While parsing a terminal animation is possible, the resulting code is going to be very brittle and will likely break if the details of that animation change. Since animations are usually not intended for machine consumption, the details may very well change.

Second, if you're trying to capture the animation - for example to replay it, or to pick it apart to see how it works - then you can run the command with a pseudo-TTY (PTY) connected to its stderr. A full breakdown of how TTYs and PTYs work is beyond the scope of an URLO post - I'd start here and follow the citations outwards, personally - but the system calls for creating and manipulating PTYs are exposed by the nix crate. Since both halves of a PTY are represented by file handles, they can be used with ordinary file APIs, including being passed to subprocesses and read from in your own process, though the bookkeeping is a bit of a pain.

3 Likes

Thank you both for your answers!!

You can check nix's behaviour pretty easily by running nix …options… |& cat

Dang it. You are right. Doing that I get 1:1 the output I am seeing from my run_filtered version, minus the filtering, of course. Looking back I should have known - the colors are all stripped away as well.

Second, if you're trying to capture the animation - for example to replay it, or to pick it apart to see how it works - then you can run the command with a pseudo-TTY (PTY) connected to its stderr. A full breakdown of how TTYs and PTYs work is beyond the scope of an URLO post - I'd start here and follow the citations outwards, personally - but the system calls for creating and manipulating PTYs are exposed by the nix crate. Since both halves of a PTY are represented by file handles, they can be used with ordinary file APIs, including being passed to subprocesses and read from in your own process, though the bookkeeping is a bit of a pain.

I'll try and see if I can get this to work through the nix crate and a PTY. Thanks for pointing me in that direction!!

It does this so that you can use tools like grep and less to read its actual output, without having to disentangle it from the auxilliary status information. I'd even argue that this is a reasonable and correct use of stderr.

OK, I can see the logic behind that, at least almost. Because actual errors still also get sent to stderr, meaning you have the same problem greping those arguably more important things now, having now mixed the "irrelevant" animated parts with the most important (error) messages, with no nice way to disentangle them.