I am learning to use std::process to spawn child processes and talk to them with pipe.
Here is the code:
use std::io::{Read, Write};
use std::process::{Command, Stdio};
fn main() {
let mut child = Command::new("rev")
// Allocate two pipes for stdin and stdout through which the parent can
// talk to the child.
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("failed to run rev");
let mut stdin = child.stdin.take().unwrap();
// Write to the child's stdin.
let _ = stdin.write_all("hello world".as_bytes());
let mut stdout = child.stdout.take().unwrap();
let mut buf = Vec::new();
// Read from the child's stdout.
let _ = stdout.read_to_end(&mut buf);
println!(
"You piped in the reverse of: {}",
String::from_utf8_lossy(&buf[..])
);
}
I expect the code to output You piped in the reverse of: dlrow olleh, but the program blocks on let _ = stdout.read_to_end(&mut buf);. Seems like the child blocks on reading from stdin because if I explicitly drop the stdin handle from parent process after writing to it, i.e.
let _ = stdin.write_all("hello world".as_bytes());
drop(stdin);
The program successfully finishes and outputs "You piped in the reverse of: dlrow olleh".
But why does the child block on the first place? Why is it able to read "hello world" until we close the pipe? stdin.write_all eventually calls libc::write which in my understanding is an unbuffered write so the child should be able to read from the pipe.
Without testing my hypotheses, I'd guess that rev very much is able to read the "hello world" before you drop the stdin. However it cannot start outputting anything before finding the first line break (or alternatively EOF) in the stdin.
And further, the call read_to_end doesn't wait for just any output to be present but indeed for EOF on stdout, and of course rev cannot end the output stream before it has finished consuming its input, i. e. receiving EOF there. This is, I suppose, why writing "hello world\n" doesn't make much of an immediate difference either.
The behavior is similar to how when running rev yourself, you won’t see any output before pressing enter for a newline
use std::io::{Read, Write};
use std::process::{Command, Stdio};
fn main() {
let mut child = Command::new("rev")
// Allocate two pipes for stdin and stdout through which the parent can
// talk to the child.
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("failed to run rev");
let mut stdin = child.stdin.take().unwrap();
// Write to the child's stdin.
stdin.write_all(&[b'\n'; 4097]).unwrap();
let stdout = (child.stdout.take().unwrap());
for l in stdout.bytes() {
println!("{}", l.unwrap());
}
}
this seems to reveals that there must also be some sort of 4kb-sized buffering involved somewhere; if I change the write_all to use 4096, it blocks before printing anything). (Calling flush() on stdin doesn’t seem to make a difference.) I wonder if this is something that Rust’s process APIs do, or something else… maybe somehow rev itself (though how then is it that in the terminal it can show the first line’s output immediately?)
Thank you for pointing out that for rev to start processing without closing the pipe, I need to write a newline at the end; and I should have used stdout.read. So this is the new code:
use std::io::{Read, Write};
use std::process::{Command, Stdio};
fn main() {
let mut child = Command::new("rev")
// Allocate two pipes for stdin and stdout through which the parent can
// talk to the child.
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("failed to run rev");
let mut stdin = child.stdin.take().unwrap();
// Write to the child's stdin.
let _ = stdin.write_all("hello world\n".as_bytes());
// drop(stdin);
let mut stdout = child.stdout.take().unwrap();
// let mut buf = Vec::new();
let mut buf = [0; 16];
// Read from the child's stdout.
// let _ = stdout.read_to_end(&mut buf);
let b = stdout.read(&mut buf).expect("failed to read");
println!(
"You piped in the reverse of: {}",
// String::from_utf8_lossy(&buf[..])
String::from_utf8_lossy(&buf[..b])
);
}
However, the program still blocks and produces no output.
I see a bunch of 10s printed starting from the 4097 size. (After they are printed, it of course still blocks). Maybe it's platform dependent, have you tried a larger number?
I use os_pipe to reproduce the scenario with this code:
use std::io::{prelude::*, Result};
use std::process::Command;
fn main() -> Result<()> {
let (reader1, mut writer1) = os_pipe::pipe()?;
let (mut reader2, writer2) = os_pipe::pipe()?;
let _child = Command::new("rev")
.stdin(reader1)
.stdout(writer2)
.spawn()
.expect("failed to run rev");
let _ = writer1
.write_all(&[b'\n'; 4096 * 4])
.expect("failed to write");
let mut output = [0; 16];
let b = reader2.read(&mut output).expect("failed to read");
println!("output: {:?}", &output[..b]);
Ok(())
}
reader2.read still blocks unless writer1 writes more than 4096*4 bytes. However this sample code provided by os_pipe's doc does not block:
use std::io::prelude::*;
let (mut reader, mut writer) = os_pipe::pipe()?;
// XXX: If this write blocks, we'll never get to the read.
writer.write_all(b"x")?;
let mut output = [0];
reader.read_exact(&mut output)?;
assert_eq!(b"x", &output);
P.S. This sample code is contradictory with description from the doc:
Pipe reads will block waiting for input as long as there’s at least one writer still open.
This seems relevant, I tested the cat command instead of rev, for comparison, and it doesn’t ever block. So maybe this is somehow behavior of rev? My working theory is that it might behave differently when spawned from a process vs when called by a user from terminal.
The buffering mentioned in man unbuffer seems very relevant.
Edit: It seems to be the case that
let mut child = Command::new("unbuffer")
.args(["-p", "rev"])
.…;
or
let mut child = Command::new("stdbuf")
.args(["-oL", "rev"])
.…;
makes the thing work. Regarding your version of the code that uses stdout.read(&mut buf) directly, of course that’s still buggy as a single call to read does not guarantee how much data will actually be read.
The fact that stdbuf -oL fixes the problem suggests that rev uses libc so that (which is default behavior as far as I understand) its stdout buffering strategy depends on whether or not its output is connected to a terminal, and that the behavior of this stdout in particular is the issue here.
Pipes can buffer a certain amount of bytes before writing to them blocks. You can change the amount using fcntl(fd, F_SETPIPE_SZ, size) to anything between the page size and the value in /proc/sys/fs/pipe-max-size.
This is the perfect answer! Indeed the inability to read from rev's write-end pipe is due to its own special behavior. I changed rev to cat and is able to read from the write end without blocking. I also tested with unbuffer and stdbuf with success.
There is also a catch, when use unbuffer to prevent buffering on write, and extra \r is written before \n.
