Broken pipe when attempt to write to stdout

Hi everyone, I am new to the language and trying to learn by doing some exersices.

I have a programA that continously writes to stdout lines and I want to pipe it to the stdin of my programB, process the line and write it to stdout. I do not have control over programA. When I try to use stdout my program panics with "Broken pipe (os error 32)" error.

fn main() {
    let stdin = io::stdin();
    let mut stdout = io::stdout().lock();
    let mut traces_by_id: HashMap<String, Trace> = HashMap::new();
    let mut identified_traces: Vec<Trace> = vec![];
    let mut buffer: Vec<String> = vec![];

    for line in stdin.lock().lines() {
        match line {
            Ok(ln) => {
                buffer.push(ln);

                if buffer.len() >= 10000 {
                    for ln in buffer.drain(..) {
                        let new_identified_traces = prune_old_traces(&mut traces_by_id);
                        identified_traces.extend(new_identified_traces);

                        // Use the parse_line function to parse the read line
                        if let Some(log_entry) = parse_line(&ln) {
                            let trace_id = &log_entry.trace_id;

                            match traces_by_id.entry(trace_id.clone()) {
                                std::collections::hash_map::Entry::Vacant(e) => {
                                    let trace = Trace {
                                        trace_id: trace_id.clone(),
                                        log_entries: vec![log_entry.clone()],
                                        lines_since_last_found: 0,
                                    };
                                    e.insert(trace);
                                }
                                std::collections::hash_map::Entry::Occupied(mut e) => {
                                    e.get_mut().lines_since_last_found = 0;
                                    e.get_mut().log_entries.push(log_entry.clone());
                                }
                            }
                        }

                        for trace in identified_traces.clone() {
                            if let Some(t) = construct_tree(trace) {
                                let output = format!("{:?}\n", t);
                                match stdout.write_all(output.as_bytes()) {
                                    Ok(_) => {}, // Write was successful, do nothing
                                    Err(e) => {
                                        eprintln!("Error writing to stdout: {}", e);
                                        continue; // Skip the current iteration and move on to the next line
                                    }
                                }
                            }
                        }
                    }
                }
            }
            Err(_) => {
                eprintln!("Error reading line");
            }
        }
    }
}

How can I write to stdout without causing panic?

A pipe to your stdin shouldn't cause your stdout to fail, so there must be something further going on, and ā€œBroken pipeā€ errors only occur when reading or writing a pipe. More information would help. Please copy the text from your shell (terminal) session including:

  • the command you are running in order to start programA and programB, and
  • the output including the panic message.

Don't edit or paraphrase it ā€” quote the text exactly. This will contain many clues about what is going wrong.

2 Likes

The producer is a program (lets call it programA) that accepts the programB executable as argument

./traces-producer ./target/debug/logs_processor

My program should "pause" the stdin, process some lines and write to stdout (from where the traces-producer checks if they are correct) and resume the stdin.

I know how to do it in NodeJS, and was able to execute it with

process.stdin.pause() and process.stdin.resume()

but in Rust there is no equivalent on pausing the stdin, or a way to signalise to the traces-producer to stop producing so my program can write to stdout.

The producer is a program (lets call it programA) that accepts the programB executable as argument ā€¦ the traces-producer checks if they are correct

Okay, so your program also has a pipe on its stdout. That makes more sense. So, if you are getting ā€œbroken pipeā€ when you write to stdout, that means the other end of the pipe was closed. Presumably your process's programA parent process has decided, for some reason, that it is done with your output already. Perhaps you wrote incorrect data, or perhaps you wrote extra data after the end of whatever protocol this is.

In any case, the right thing to do upon ā€œbroken pipeā€ is to report the error and exit. Your program has no further responsibilities, because its parent has either decided it is no longer needed, or made a mistake. Treat the ā€œbroken pipeā€ not as a thing to be fixed or dealt with itself, but as a sign that something preceding that point went wrong.

I know how to do it in NodeJS, and was able to execute it with

process.stdin.pause() and process.stdin.resume()

If you don't want more data at the moment, simply don't read from the pipe. There is no pause because there are no asynchronous data-received events; refraining from reading has the same effect as far as other processes are concerned.

Probably relevant, if you want to handle the error in a more graceful or typical way:

2 Likes

Exit yes. But you don't want to report a broken pipe error. Typical Unix CLI applications will actually receive a signal on broken pipe and then unceremoniously exit. That's what, for example, tools piped into head and grep will do. The Rust runtime does things differently and ignores SIGPIPE, which results in broken pipe errors occurring as "normal" errors to read. If you treat them like any other error, then your CLI program won't match the Unix convention. (And this isn't just matching convention for the sake of it. If your Rust program prints an error because of a broken pipe caused by head -n1, that's not good.)

This is why println! is a footgun in production code.

1 Like

I see. I was incorrectly thinking that an error was appropriate given that the program won't be exiting with a "terminated due to signal" status, and not considering the situation around head or similar.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.