Syn, how to peek at `token.to_string()`?

I can parse_stream.peek(Ident) to see if the next token is an identifier, but can I peek to see if the next token is an identifier named "void"?

Use case: I'm trying to parse old-style C function signatures which may contain the C keyword void (e.g., closeall(void) or close(int fd)), so I'd like to do something like this:

let void: Option<Token![void]>;
let arg_types: Punctuated<ArgType, Token![,]>;
if input.peek(Token![void]) && input.peek2(Token![)]) {
    void = input.parse()?;
    arg_types = Punctuated::new()
} else {
    void = None;
    arg_types = content.parse_terminated(ArgType::parse, Token![,])?
};

But of course, void is not a keyword and is not recognized by Token![...]. I would need to replace that with Ident and also have some way of testing the value of the ident.

Ident is PartialEq<S> where S: AsRef<str>.


By the way, for writing custom parsers over token streams, try my crate parsel. It lets you define structured AST nodes that parse themselves, and it also lets you define custom keywords.

How do I get an instance of Ident, test it (ok, you've shown me how to do that), and if the test fails, "unparse" the token so that it can be parsed another way?

Or, how do I "peek" at the next token as an object, to test it with the string, in a way that doesn't consume it from the parse stream (see my above code)?

I understand ParseBuffer::fork may work, but that seems pretty discouraged. The docs say fork is usually for parsing (A* B*) (source), but I just need to parse two options (A | B), which I was hoping to do with a peek that dispatches on the value of the token--not just the type.

Maybe this helps? custom_keyword in syn - Rust

1 Like

Also note that parentheses are not tokens, so this won’t work, either. For C function signatures, there are non parameter-lists starting with void besides the whole thing being (void), are there? So maybe a input.peek(kw::void)[1] may be enough; the parsing infrastructure should already make sure that no further tokens are left unconsumed, as far as I understand (I didn’t test this assumption).

Otherwise, you could work with input.cursor() directly to check whether it’s having eof after consuming one ident.


  1. with kw::void as defined via custom_keyword ↩︎

fork seems appropriate here. And yes, custom_keyword to look for "void" as an identifier. The way I would write it is something like this:

use syn::parse::discouraged::Speculative;
use syn::parse::{Parse, ParseStream, Result};
use syn::punctuated::Punctuated;
use syn::{parenthesized, Ident, Token};

mod kw {
    syn::custom_keyword!(void);
}

enum Sig {
    Void,
    NonVoid(Punctuated<ArgType, Token![,]>),
}

impl Parse for Sig {
    fn parse(input: ParseStream) -> Result<Self> {
        let _name: Ident = input.parse()?;

        let args;
        parenthesized!(args in input);

        let ahead = args.fork();
        let void: Option<kw::void> = ahead.parse()?;
        if void.is_some() && ahead.is_empty() {
            args.advance_to(&ahead);
            Ok(Sig::Void)
        } else {
            let args = args.parse_terminated(ArgType::parse, Token![,])?;
            Ok(Sig::NonVoid(args))
        }
    }
}
2 Likes

For clarification, the docs for Speculative::advance_to talk about A* B* because that's a common grammar construction that ends up requiring un-/loosely bounded speculative parsing (or making a messy manual NFA to DFA transform) to parse, not because it's the only case where it's useful. The docs for ParseBuffer::fork talk about what's actually a very analogous syntactical case with pub(self).

When you have an existing grammar to match, you parse it however you need to. When you're defining a new grammar, sticking to LL(3) peek lookahead is preferable.

I don't see any other way. Unfortunately, peek() returns bool, not Option<Token>, and it's impossible to implement Peek for your own custom type, since the trait is sealed. (The FnOnce(TokenMarker) -> T blanket impl doesn't apply either, becaues TokenMarker is private too.)