Communication with child process through pipe

Hi there,

I am learning to use std::process to spawn child processes and talk to them with pipe.

Here is the code:

use std::io::{Read, Write};
use std::process::{Command, Stdio};

fn main() {
    let mut child = Command::new("rev")
        // Allocate two pipes for stdin and stdout through which the parent can
        // talk to the child.
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .spawn()
        .expect("failed to run rev");

    let mut stdin = child.stdin.take().unwrap();
    // Write to the child's stdin.
    let _ = stdin.write_all("hello world".as_bytes());

    let mut stdout = child.stdout.take().unwrap();

    let mut buf = Vec::new();
    // Read from the child's stdout.
    let _ = stdout.read_to_end(&mut buf);

    println!(
        "You piped in the reverse of: {}",
        String::from_utf8_lossy(&buf[..])
    );
}

I expect the code to output You piped in the reverse of: dlrow olleh, but the program blocks on let _ = stdout.read_to_end(&mut buf);. Seems like the child blocks on reading from stdin because if I explicitly drop the stdin handle from parent process after writing to it, i.e.

let _ = stdin.write_all("hello world".as_bytes());
drop(stdin);

The program successfully finishes and outputs "You piped in the reverse of: dlrow olleh".

But why does the child block on the first place? Why is it able to read "hello world" until we close the pipe? stdin.write_all eventually calls libc::write which in my understanding is an unbuffered write so the child should be able to read from the pipe.

Without testing my hypotheses, I'd guess that rev very much is able to read the "hello world" before you drop the stdin. However it cannot start outputting anything before finding the first line break (or alternatively EOF) in the stdin.

And further, the call read_to_end doesn't wait for just any output to be present but indeed for EOF on stdout, and of course rev cannot end the output stream before it has finished consuming its input, i. e. receiving EOF there. This is, I suppose, why writing "hello world\n" doesn't make much of an immediate difference either.

The behavior is similar to how when running rev yourself, you won’t see any output before pressing enter for a newline


and even after doing that, you’ll see the current line reversed and outputted

but you don’t get back into your shell prompt (i.e. out of rev) before ending the input stream (via Ctrl+D on Linux).

3 Likes

Experimenting with this a little bit…

use std::io::{Read, Write};
use std::process::{Command, Stdio};

fn main() {
    let mut child = Command::new("rev")
        // Allocate two pipes for stdin and stdout through which the parent can
        // talk to the child.
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .spawn()
        .expect("failed to run rev");

    let mut stdin = child.stdin.take().unwrap();
    // Write to the child's stdin.
    stdin.write_all(&[b'\n'; 4097]).unwrap();

    let stdout = (child.stdout.take().unwrap());

    for l in stdout.bytes() {
        println!("{}", l.unwrap());
    }
}

this seems to reveals that there must also be some sort of 4kb-sized buffering involved somewhere; if I change the write_all to use 4096, it blocks before printing anything). (Calling flush() on stdin doesn’t seem to make a difference.) I wonder if this is something that Rust’s process APIs do, or something else… maybe somehow rev itself (though how then is it that in the terminal it can show the first line’s output immediately?)

Thank you for pointing out that for rev to start processing without closing the pipe, I need to write a newline at the end; and I should have used stdout.read. So this is the new code:

use std::io::{Read, Write};
use std::process::{Command, Stdio};

fn main() {
    let mut child = Command::new("rev")
        // Allocate two pipes for stdin and stdout through which the parent can
        // talk to the child.
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .spawn()
        .expect("failed to run rev");

    let mut stdin = child.stdin.take().unwrap();
    // Write to the child's stdin.
    let _ = stdin.write_all("hello world\n".as_bytes());
    // drop(stdin);

    let mut stdout = child.stdout.take().unwrap();

    // let mut buf = Vec::new();
    let mut buf = [0; 16];
    // Read from the child's stdout.
    // let _ = stdout.read_to_end(&mut buf);
    let b = stdout.read(&mut buf).expect("failed to read");

    println!(
        "You piped in the reverse of: {}",
        // String::from_utf8_lossy(&buf[..])
        String::from_utf8_lossy(&buf[..b])
    );
}

However, the program still blocks and produces no output.

Right, that's what I pointed out in my second reply, too. I have no idea yet where exactly the apparent buffering here comes from :slight_smile:

This piece of code also blocks on my computer. When you write 4097 \n, do you see them output to stdout?

P.S. Sadly flush stdin does nothing. Eventually stdin.flush simply returns Ok(()).

impl Write for &ChildStdin {
    fn flush(&mut self) -> io::Result<()> {
        Ok(())
    }
}

I see a bunch of 10s printed starting from the 4097 size. (After they are printed, it of course still blocks). Maybe it's platform dependent, have you tried a larger number?

I had to bump the number of newline to 4096*4+1 before I can see a bunch of 10s output to termianl. The toolchain is stable-x86_64-apple-darwin.

You could try if using os_pipe - Rust can change the behavior.

I use os_pipe to reproduce the scenario with this code:

use std::io::{prelude::*, Result};
use std::process::Command;

fn main() -> Result<()> {
    let (reader1, mut writer1) = os_pipe::pipe()?;

    let (mut reader2, writer2) = os_pipe::pipe()?;
    let _child = Command::new("rev")
        .stdin(reader1)
        .stdout(writer2)
        .spawn()
        .expect("failed to run rev");

    let _ = writer1
        .write_all(&[b'\n'; 4096 * 4])
        .expect("failed to write");

    let mut output = [0; 16];
    let b = reader2.read(&mut output).expect("failed to read");

    println!("output: {:?}", &output[..b]);

    Ok(())
}

reader2.read still blocks unless writer1 writes more than 4096*4 bytes. However this sample code provided by os_pipe's doc does not block:

use std::io::prelude::*;

let (mut reader, mut writer) = os_pipe::pipe()?;
// XXX: If this write blocks, we'll never get to the read.
writer.write_all(b"x")?;
let mut output = [0];
reader.read_exact(&mut output)?;
assert_eq!(b"x", &output);

P.S. This sample code is contradictory with description from the doc:

Pipe reads will block waiting for input as long as there’s at least one writer still open.

This seems relevant, I tested the cat command instead of rev, for comparison, and it doesn’t ever block. So maybe this is somehow behavior of rev? My working theory is that it might behave differently when spawned from a process vs when called by a user from terminal.

The buffering mentioned in man unbuffer seems very relevant.

Edit: It seems to be the case that

    let mut child = Command::new("unbuffer")
        .args(["-p", "rev"])
        .…;

or

    let mut child = Command::new("stdbuf")
        .args(["-oL", "rev"])
        .…;

(what are these?)

makes the thing work. Regarding your version of the code that uses stdout.read(&mut buf) directly, of course that’s still buggy as a single call to read does not guarantee how much data will actually be read.

The fact that stdbuf -oL fixes the problem suggests that rev uses libc so that (which is default behavior as far as I understand) its stdout buffering strategy depends on whether or not its output is connected to a terminal, and that the behavior of this stdout in particular is the issue here.

1 Like

Pipes can buffer a certain amount of bytes before writing to them blocks. You can change the amount using fcntl(fd, F_SETPIPE_SZ, size) to anything between the page size and the value in /proc/sys/fs/pipe-max-size.

1 Like

This is the perfect answer! Indeed the inability to read from rev's write-end pipe is due to its own special behavior. I changed rev to cat and is able to read from the write end without blocking. I also tested with unbuffer and stdbuf with success.

There is also a catch, when use unbuffer to prevent buffering on write, and extra \r is written before \n.

So, in the previous example, the buffer of a pipe is set to 16 KB. Excellent!

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.