Syn 0.15 -- improved parsing API that can trigger errors in the right place on invalid macro input

announcements
1

Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.

https://github.com/dtolnay/syn

This release introduces a new parsing API that enables procedural macros to easily report intelligent errors when provided syntactically invalid macro input.

This functionality is important because Rust 1.29.0, shipping next week, will be stabilizing support for defining functionlike!(...) procedural macros. Until now procedural macros have only been usable as custom derives, for which the compiler guarantees a syntactically valid input and thus error reporting is not the responsibility of the macro. Function-like macros a.k.a. "bang" macros are different in that the caller may pass any invalid crazy input and needs to be told by the macro where they messed it up.

See below a few examples of error messages produced by Syn with almost no effort or attention to error handling on the part of the macro author.

Parsing

With the transition to the new parsing API we drop the use of nom-style parser combinator macros in favor of plain Rust functions and control flow. Parsers should be easier to write, much easier to read, and emit reasonable error messages with very little effort.

Here is how we might implement parsing for a simplified representation of Rust structs.

/// A struct with named fields, like `struct S { a: u8, b: String }`
struct Struct {
    struct_token: Token![struct],
    ident: Ident,
    brace_token: token::Brace,
    fields: Punctuated<Field, Token![,]>,
}

impl Parse for Struct {
    fn parse(input: ParseStream) -> Result<Self> {
        let content;
        Ok(Struct {
            struct_token: input.parse()?,
            ident: input.parse()?,
            brace_token: braced!(content in input),
            fields: content.parse_terminated(Field::parse_named)?,
        })
    }
}

Another example, input that may be either a struct or an enum:

enum Item {
    Struct(Struct),
    Enum(Enum),
}

impl Parse for Item {
    fn parse(input: ParseStream) -> Result<Self> {
        let lookahead = input.lookahead1();
        if lookahead.peek(Token![struct]) {
            input.parse().map(Item::Struct)
        } else if lookahead.peek(Token![enum]) {
            input.parse().map(Item::Enum)
        } else {
            Err(lookahead.error())
        }
    }
}

This release introduces a parse_macro_input! macro that handles shuffling errors correctly back to the compiler inside of a procedural macro.

#[proc_macro]
pub fn my_macro(tokens: TokenStream) -> TokenStream {
    let input = parse_macro_input!(tokens as Item);

    /* `input` is of type Item, our data structure defined above */
}

Let's try feeding this macro some invalid input. Syn triggers a compiler error indicating the problematic token and information about what the parser would have expected in that position where possible.

Note: the APIs necessary for Syn to display these errors in the right place have been stabilized in the compiler but are still riding the release trains, will be landing in the stable compiler in 1.29.0 next week. On older stable compilers the error message will be the same as shown here but the position will be wrong.

error: expected `struct` or `enum`
 --> src/main.rs:7:11
  |
7 | my_macro!(uh-oh);
  |           ^^

error: expected `:`
 --> src/main.rs:7:31
  |
7 | my_macro!(struct S { a: u8, b String });
  |                               ^^^^^^

error: unexpected token
 --> src/main.rs:7:30
  |
7 | my_macro!(struct S { a: u8 } @);
  |                              ^

Compile time and performance

As always, much attention has been paid to compile time. Error reporting is nice but we don't want to spend an excessive amount of time compiling tons of error handling code.

The parsing component has been designed with compile time in mind and Syn 0.15 consistently compiles in around the same amount of time as the most recent point release of 0.14. This is despite adding support for a variety of new Rust syntax accepted through RFCs and eRFCs in recent months.

Parsing performance improves by 50% with the move to the new parsing API, as measured by time taking to parse every Rust source file in the rust-lang/rust repo and its submodules. That time is down from 10.6 seconds to 6.9 seconds on my machine.

Newly supported Rust syntax

Syn 0.15 documentation on docs.rs

11 Likes