Number parsers that 'try', like strtoll

1

I am looking for numeric parsers that incrementally consume their input until they hit a mismatch, this is the behavior of strtol in the c standard, it eats as many characters as it can and then returns the value it parsed plus a pointer to where it gave up.

I am writing a lexer that needs to see if what comes next is a float or integer constant

2

One could use s.split_once(|ch| !ch.is_numeric())?.0 to get the leading part of the string that's a valid integer; to add floats, add ... || *ch == '.' to the end of the condition, then check to see if the returned substring contains a decimal point or not. After that, you can parse using the regular Rust methods.

3

I haven't used it myself, but have heard good things about nom. It uses a pattern like fn parse(&str) -> IResult<&str, T>, where (on success) you get back the parsed value and the remaining part of the input. I'm not sure how well this aligns with your use case.

1 Like
4

If you're using iterators to parse the tokens, you can just iterate over the items one-by-one and in the error case return a new iterator via std::iter::once(failed_item).chain(remainder_of_original_iterator), similar to what I did here:

5

problem is that float constants are very complicated beasts, tryng to avoid parsing them myself. Pure laziness

6

Different languages have different supported syntaxes for floating point once you get into the weeds. If you're writing a lexer, you probably want to pick what you support, but it's not too much work to match the standard format:

self.next_if('+') || self.next_if('-');
self.digits();
if self.next_if('.') {
  self.digits();
}
if self.next_if('e') || self.next_if('E') {
  self.next_if('+') || self.next_if('-');
  self.digits();
}