Pattern match line-by-line in macro_rules?

I am working on a command line library for rust . Right now it can run command like this : run_cmd!(du -ah . | sort -hr | head -n 10); by doing pattern match trick in macro_rules to filter "-" and "--" from token stream. However, I can not use the same trick to implement something like this:

group_cmds! { // return Err(...) if any command fails

Is there a way to do pattern match line-by-line in macro_rules?

AFAIK, the line separator between tokens in macro is treated the same as would any other whitespace. So, you perhaps should group tokens in each line somehow?

I would use semicolons to separate the different commands. That skips the problem around macros counting newlines as just another whitespace character, plus it's valid bash syntax.

group_cmds! { 
1 Like

Thanks for the replies. yeah, I finally made it work, which is similar to my previous solution. The problem is that ';' is also part of invisible token, so I still need use some pattern match trick to make macro_rules happy.

This link has some tricks to do pattern repetitions, however I can only print the commands, not being able to run with another macro.

If you really intend on offering a nice syntax that is hard to parse with macro_rules! macros, I suggest you transition to using procedural macros

Setting up a procedural macro crate

First of all, a proc_macro crate can only export procedural macros, nothing else.
Moreover, a function-like procedural macro (i.e., those with the same calling interface as a macro_rules! macro), are not yet supported in stable Rust, unless you use ::proc-macro-hack.

Both ::proc-macro-hack and the need to sometimes export custom types or traits lead to the two-crate pattern.

The two-crate pattern

The idea is that although a proc_macro crate can only export / offer procedural macros, a regular crate depending on it can reexport these procedural macros, while also being able to export classical stuff. Hence the idea of using an internal helper crate where the proc_macros are defined, and using a "front crate" that depends on it, reexports the stuff: that's the crate users of the library will depend on.

Directory structure

For the my_super_crate respository, you can have the following directory structure:

├── Cargo.toml
├── my_super_crate
│   ├── Cargo.toml
│   └── src
│       └──
└── my_super_crate-proc_macro
    ├── Cargo.toml
    └── src
  • ./Cargo.toml
    members = ["my_super_crate", "my_super_crate-proc_macro"]



proc_macro = true

# ...

proc_macro2 = "1.0.0"
proc-macro-hack = "0.5.9"
quote = "1.0.0"
syn = { version = "1.0.0", features = [ ... ] }  # syn usually requires extra features


extern crate proc_macro;
use ::proc_macro::TokenStream;
use ::proc_macro2::{
    TokenStream as TokenStream2,
use ::syn::{*,
    parse::{Parse, Parser},
    Result,  // explicitly shadow std's Result
use ::std::*;

macro_rules! unwrap {(
) => (
    match $res {
        | ::std::result::Result::Ok(inner) => inner,
        | ::std::result::Result::Err(err) => return err.to_compile_error().into(),

struct Cmd {
    foo: Ident, // data extracted from parsing a single command
    // etc.

impl Parse for Cmd {
    fn parse (input: ParseStream<'_>) -> Result<Self>
        let foo: Ident = input.parse()?;
        // etc.
        Self {
            // etc.

#[::proc_macro_hack::proc_macro_hack] pub
fn group_cmds (input: TokenStream) -> TokenStream
    type Cmds = Punctuated<Cmd, Token![;]>; // sequence of `Cmd`s separated by `;`
    let parser = Cmds::parse_terminated; // syn parser for that
    let cmds: Vec<Cmd> = unwrap!(parser.parse(input)).into_iter().collect();
    let resulting_code = unwrap!(my_helper_function(cmds));
    TokenStream::from(quote! {

fn my_helper_function (cmds: Vec<Cmd>) -> Result<Expr>
    let thing = stuff(cmds);
    if some_bad_condition(thing) {
        return Err(Error::new(
            Span::call_site(), // are of code to be highlighted when erroring
            format!("Expected something else"),
    let forty: Expr = parse_quote! {
    let two: Expr = parse_quote! {
    let forty_two: Expr = parse_quote! {
        #forty + #two



version = "x.y.z"  # keep in sync with dependencies proc-macro

# ...

proc-macro-hack = "0.5.9"

package = "my_super_crate-proc_macro"
path = "../my_super_crate-proc_macro"
version = "x.y.z"  # keep in sync with `../my_super_crate-proc_macro/Cargo.toml`


#[::proc_macro_hack::proc_macro_hack] // attribute to reexport a function-like proc_macro
pub use ::proc_macro::group_cmds;

// stuff

Publishing the crate

Since my_super_crate depends on my_super_crate-proc_macro, the latter needs to be published before the former:

shell $ (cd my_super_crate-proc_macro && cargo publish)
shell $ (cd my_super_crate && cargo publish)


Instead of having to go through complicated recurive loops and tt munchers with a macro_rules! macro, you can go procedural and get to program a parser (c.f., the Parser trait) instead of "hacking" one.



If you do decide to go with procedural macros I found this proc-macro workshop video super helpful.

Thanks for the detailed reply, it didn't show a lot advantages for the above simple task. I will probably consider using proc macro for defining new shell functions which could simply execute a group of commands, and return either CmdResult or FunResult. Something like this:

def_fun! {
     let foo = $1;
     let bar = $2;

     cat ${foo} | grep ${bar}

However, I am not quite familiar with proc macros, so if someone could contribute implementing some ideas, it would be much appropriated.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.