How to get full output from Command?

#1

Everytime I try to use Command interface, I struggle with this, and I never understand how is it supposed to work. I want to run a command, ignore all other stuff and just get its output. There's a method for exactly this:

String::from_utf8(
   Command::new("rg")
        .arg("\"pattern\"")
        .arg("../")
        .arg("--files-with-matches")
        .output()
        .unwrap().stdout)
.unwrap()

But it doesn't actually work as expected. For some reason, this only returns first line. Ok, so in the docs, it says that any attempt to read from stdin will cause the process to end. But no matter what I tried - all combinations of .stdin|out|err(piped() or null()) with spawn, read, read_to_string, wait_with_output, etc.. nothing works, it always returns just the first line. And it's not even clear if read from stdin is actually the cause of this problem. After two hours of hopeless trial and error, I found out that this works though:

String::from_utf8(
   Command::new("sh")
        .arg("-c")
        .arg("rg \"pattern\" ../ --files-with-matches")
        .output()
        .unwrap().stdout)
.unwrap()

But I have no idea why. Can somone point me to where might be a problem?

#2

This should return whole output. My guess would be that the output is only a single line long?

Note that, in the first version, quotes are included in the pattern, so its quite likely that there's only single maching file -- the source code itself. You probably want

let output =  Command::new("rg")
        .arg("pattern")
        .arg("../")
        .arg("--files-with-matches")
        .output()
        .unwrap();
let stdout = String::from_utf8(output.stdout).unwrap()

In general, the best way to debug such issues is not to tweak stuff until it works, but to remove and simplify stuff until it works. The nice simplification step here would be Comman::new("ls").output().

EDIT: shameless plug: xshell crate makes calling external processes slightly less annoying.

2 Likes
#3

This should return whole output.

The thing is - it usually does. I use this for some regression tests and it works fine, that's why I was surprised that it doesn't work in this case. The example I gave is simplified - In the real code, I composed the pattern programmaticaly:

    let collector = Command::new("rg")
        .arg((declarations.iter().map(|decl| {
            return String::from("(") + decl.name + ")";
        }).fold(String::from("\"(_______)"), |acc, name| {
            acc + "|" + &name
        }) + "\"").as_str())
        .arg("../")
        .arg("--files-with-matches")
        .output()
        .unwrap()
        .stdout;

I tried to print the generated arguments and use the command manually, it should've print 3 lines. Maybe there's a misquote or something that I don't see...

Note that, in the first version, quotes are included in the pattern

This is maybe the bit I am missing. Does it mean, that quoting the pattern is a terminal thing, and when you do it in terminal, rg will recieve the argument with quotes stripped? Meanwile, using Command API won't go through terminal, but instead puts the arguments directly into the program, with quotes included? That would actually explain the problem (and maybe even multiple similar problems I was struggling with in the past), even though it's a strange coincidence that the pattern actually matched something at all.

And thanks, I didn't know about xshell and it looks super cool. I actually thought about a tool like this before, It's great to see it exists.

#4

Yes, quoting is used by your shell to determine how to split the command line into arguments. It then passes the program the un-quoted, un-escaped versions of those arguments.

Shell quoting and escaping aren't needed or supported when calling APIs like Command directly, because the arguments are already split into separate strings.

2 Likes
#5

I think the way to verify this is to do

let mut cmd = Command::new("rg");
cmd.arg("pattern")
        .arg("../")
        .arg("--files-with-matches");
println!("{:?}", cmd);

And then paste the output into the shell, with all the quotes.

1 Like
#6

Let me add to the excellent advice provided by others. If you need ripgrep functionality, please note that you don't need to (and therefore shouldn't) use it by executing the command-line binary. Ripgrep provides a library crate called grep, with an API that you can use directly from code.

1 Like
#7

@mbrubeck thanks for the explanation. Do you think it'd be worth adding this to docs for .arg() and .args()? I have already been bitten by this multiple times (in multiple languages actually). Maybe it's a common knowledge but I'd certainly appreciated it in the docs. I can make a PR.

@matklad Good point, I'll try that.

@H2CO3 Thanks, I didn't know about the library :wink: But for this program, it's probably not worth it. It's just a script for some refactoring that I wrote yesterday and I'll throw it away tomorrow when I'm done, so it doesn't really matter. It's not really worth the compile-time hit for a whole workspace and there's no need learn new api, when I already know rg and Command API (maybe it'd would have been faster than debugging this quotation issue, though :smile: )

#8

I think it would be nice, personally. I also only discovered that shells work this way through using Rust's Command API; among many other such details I hadn't considered before using Rust. The docs for Command::arg kind of hint at it by saying you can only pass one argument per use, but I think it would be possible to clear it up a bit without adding too much text.

#9

It should be noted that the particular details of argument passing behaviour is OS (or even shell) specific. So the docs have to tread carefully.

It might be good to put more emphasis on "pass one argument per use" but it also needs to avoid describing implementation details that may not be universal.

#10

I opened a docs PR https://github.com/rust-lang/rust/pull/78599

2 Likes