Iterating & executing Commands

Hey,
new to rust, so I'm playing around with it :slight_smile:

I have a vec of Strings that represent shell commands. I want to iterate over them and execute them one by one. While I have "something" running, I'm sure a) it's not idiomatic and b) it's not working for every case (see the last two, where I'm redirecting output/piping).

How can I improve my code?

use std::process::Command;

fn main() {
    let commands = vec![
        "ls -lah".to_string(),
        "echo 'this is a really long string'".to_string(),
        "echo 'hello world > file.txt".to_string(),
        "cat file.txt | grep hello".to_string(),
    ];

    let commands_iterator = commands.into_iter();

    commands_iterator.for_each(|cmd| {
        println!("---------");
        println!("{}", cmd);

        let mut cmd_parts = cmd.split(" ");

        let mut c = Command::new("");

        if let Some(a) = cmd_parts.next() {
            println!("{:?}", a);
            c = Command::new(a);
        }

        cmd_parts.for_each(|part| {
            println!("> {}", part);
            c.arg(part);
        });

        c.status().expect("failed to execute process");
    });
}

The output:

---------
ls -lah
"ls"
> -lah
total 20K
drwxr-xr-x 1 runner runner 104 May 19 08:38 .
drwxrwxrwx 1 runner runner  68 May 19 08:00 ..
drwxr-xr-x 1 runner runner  12 Oct  8  2021 .cache
-rw-r--r-- 1 runner runner 154 Oct 16  2021 Cargo.lock
-rw-r--r-- 1 runner runner 200 Dec 17  2021 Cargo.toml
-rw-r--r-- 1 runner runner 144 Apr 23 18:29 .replit
-rw-r--r-- 1 runner runner  25 Apr 24 14:20 replit.nix
drwxr-xr-x 1 runner runner  14 May 19 08:39 src
drwxr-xr-x 1 runner runner  66 Apr 12 23:55 target
---------
echo 'this is a really long string'
"echo"
> 'this
> is
> a
> really
> long
> string'
'this is a really long string'
---------
echo 'hello world > file.txt
"echo"
> 'hello
> world
> >
> file.txt
'hello world > file.txt
---------
cat file.txt | grep hello
"cat"
> file.txt
> |
> grep
> hello
cat: file.txt: No such file or directory
cat: '|': No such file or directory
cat: grep: No such file or directory
cat: hello: No such file or directory

Any help appreciated :slight_smile:

The code has several problems but none of them are really specific to Rust.

Command literally executes commands (i.e., executables); it can't perform shell expansion/interpolation. You shouldn't be doing this, in any case – shell expansion is not as simple as splitting on white space, and trying to perform naΓ―ve "string manipulation" on what really is an abstract syntax tree for a programming language won't end well (e.g. it makes the resulting program vulnerable to code injection more often than not).

If your only requirement is that you want to pipe the stdout of one program to the stdin of the next one, then follow the official documentation (the gist is using Stdio::piped() and Stdio::from() at the appropriate places).

Do not represent your programs as a string to be split on whitespace; that will fail to correctly handle arguments with whitespace in them (incidentally, you think your 2nd echo command works, but it doesn't, for this exact reason – it prints every word on a new line). Instead, store commands as slices/vectors/etc. of arguments, one item for each argument (including the executable).

If you really want to execute shell commands, you'll have to call the shell explicitly, but that's strongly recommended against, as it can be abused with basically arbitrarily bad side effects.

Some Rust-specific problems in your code are unnecessarily converting your string literals to String, unnecessary mutability, and unnecessary use of .into_iter().for_each() instead of a simple for loop.

You'd be much better off creating a typed specification of your pipelines and commands, and interpreting that structured data instead of going through the shell. Here is a simple solution:

#[derive(Clone, Debug)]
struct PipelineSpec<T> {
    members: Vec<CommandSpec<T>>,
    output_file: Option<T>,
}

#[derive(Clone, Debug)]
struct CommandSpec<T> {
    executable: T,
    args: Vec<T>,
}

fn main() {
    let commands = vec![
        PipelineSpec { 
            members: vec![
                CommandSpec { executable: "ls", args: vec!["-lah"] },
            ],
            output_file: None,
        },
        PipelineSpec {
            members: vec![
                CommandSpec { executable: "echo", args: vec!["this is a really long string"] },
            ],
            output_file: None,
        },
        PipelineSpec {
            members: vec![
                CommandSpec { executable: "echo", args: vec!["hello world"] },
            ],
            output_file: Some("file.txt"),
        },
        PipelineSpec {
            members: vec![
                CommandSpec { executable: "cat", args: vec!["file.txt"] },
                CommandSpec { executable: "grep", args: vec!["hello"] },
            ],
            output_file: None,
        },
    ];

    for spec in commands {
        println!("---------");
        println!("{:?}", spec);

        let Some((first, rest)) = spec.members.split_first() else {
            println!("empty command");
            continue;
        };

        let mut child = Command::new(first.executable)
            .args(&first.args)
            .stdout(Stdio::piped())
            .spawn()
            .expect("could not spawn");

        for cmd in rest {
            let next = Command::new(cmd.executable)
                .args(&cmd.args)
                .stdout(Stdio::piped())
                .stdin(Stdio::from(child.stdout.expect("spawned child does not pipe stdout")))
                .spawn()
                .expect("could not spawn");
            child = next;
        }

        let output = child.wait_with_output().expect("error waiting on last command of pipeline");

        if let Some(path) = spec.output_file {
            fs::write(path, &output.stdout).expect("can't write output file");
        } else {
            let output_str = String::from_utf8(output.stdout).expect("invalid UTF-8");
            println!("{}", output_str);
        }
    }
}

As an aside, if you are executing cat file | other_command, you're probably doing something wrong, this is an anti-pattern – it potentially reads big files into memory and creates an unnecessary pipe, losing file metadata (this can e.g. inhibit useful progress bars). If you want to run a command on a file, then run the program directly on the file. If the program can only read from standard input, it's still better to do command < file.txt instead, as the built-in redirection can handle the file in a smarter way, usually mitigating the problems that an explicit cat causes.

7 Likes

If you want to execute shell commands, my recommendation is a crate called run_script. I have been using it for a while and it fits this exact use case.

Here it is if you are interested!
https://crates.io/crates/run_script

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.