Testing Rustc lexer

Recently, I came across the Rustc_lexer, and built something similar. The lexer is a stand alone crate for rust, and I am trying to test the lexer to see how rust interprets the character and tokens -- Im testing the lexer to see if it would be compatible for multi-characters, while also looking at the possibility of creating my own version of AI.

I need help on the following code, which runs in main.rs and uses the rustc_lexer crate. I need to put a command line utility together to run the lexer from the command line and output to the terminal. How would a rust programmer do this? Below is a snippet of the source that Im trying to use:

fn main() -> io::Result<()> {
    // Get the command line arguments
    let args: Vec<String> = env::args().collect();

    // Check if the user provided a file name
    if args.len() < 2 {
        eprintln!("Please provide a file name");
        return Ok(());

    // Open the file
    let file = File::open(&args[1])?;

    // Create a lexer to tokenize the input
    let lexer = Lexer::new(file);

    // Iterate over the tokens and print them
    for token in lexer {
        println!("{:?}", token);


Instead of using a file source. I would like to input the character strings directly in the command line, and have the terminal print out the tokens obtained from the character string.

Any help would be greatly appreciated, and most welcome. Thank you kindly!! :slight_smile:

For boilerplate-free argument parsing, use clap. For reading from standard input, use the Read impl of std::io::stdin().

Thank you kindly!

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.