I have created a very simple lexer for Lisp-like text analysis. Could you provide feedback please?

My code lives here Can you point things that I am doing wrong out or suggestions to do things more efficient or better in any way?
Thanks.

At a first glance, this seems pretty good! I don't think I have any recommendations for improving performance or correctness.

With that said, there is one thing I'd recommend adding: a few short comments at key points to explain what purpose variables serve, and what the code is doing. For instance, it took me a while to figure out why you had no_push_stack - having a short comment explaining it's purpose would have greatly sped up how fast I could have read it. And similarly, that speeds up other people understanding the code, and you understanding the code a few months from now.

If you want to strive for maximum efficiency, it might be interesting to try doing a "zero copy" parser which never allocates new strings. You could define your token type as

enum Token<'a> {
    Token(&'a str),
    Sub(Vec<Token<'a>>),
}

and then your function would return a list of tokens which all borrow from the original string. It'd be much more efficient, but also mean you have to keep the original string around.

But really, that would kind of be changing what it does - a large optimization. For the function as you've written it, comments are the only thing I think it needs. Otherwise, it looks good!

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.