_Bug_ in if let statement

[J-Cake]

Good morning all,

I'm learning about parsers and recursion in Rust and keep getting stack-overflow errors. I'm familiar enough with recursion to know that means I've screwed up somewhere, but stepping through with the debugger I've encountered a weird issue where I would destructure the result of a sub parser using an if let statement, and on Ok it would return an AST node with a guard-clause-style setup. But when the destructuring failed, the calling function would not simply continue, but would repeat the if let expression indefinitely, overflowing the stack.

Here's an example:

Here is a simplification of my code

The debugger shows the following pattern: (Note the line numbers are adjusted to fit the gist)

1. The program starts at line 19
2. Jump to line 3
3. The precondition fails - jumps to line 4
4. The if condition on line 19 fails, returns.
5. Jump to line 19
6. Repeat steps 2 - 6

This short screen recording demonstrates my point of confusion

I've checked to make sure the condition is failing etc, which it is, so I presume there is some sort of weird behaviour in the way rust destructures certain objects. Anyway, I'm not familiar enough with Rust or LLDB or GDB to continue to debug this issue further myself, I was hoping for some insights.

Any help and intellect is greatly appreciated.
Jake

[steffahn]

The screen recording appears not to be public. From the limited information, I would need to agree that it seems weird what you are observing. Of course, it’d be useful to double-check that the debugger is used correctly, and maybe also to check that the thing that makes control flow go back to line 19 is really a return, and not another recursive call.

If the source code producing this issue is open-source, it could be useful if the whole code was available, so someone else can reproduce and help properly minimize the issue.

[J-Cake]

To address all your questions in order,

1. I'm frustrated that Google needs to process the video. It's a 40s clip of me stepping through with the debugger...
   1.5 I've realised that I hadn't set visibility options correctly, it should be fixed
2. I've been scratching my head for a while, I don't think I've missed something, although that is quite possible. There's no recursive call in the failing clause etc.
3. Absolutely. The code is the most recent commit. The file in question is

Thanks again

[steffahn]

Seems to be a straightforward infinite recursion.

parse calls parse_index, which splits up the tokens in top_level_split and afterwards starts parsing the resulting token sections again from the beginning with a call to parse to each element of the split up tokens, using map(parse).

Also the if tokens.len() != 1 { return Err(…) } in that function seems suspicious, though inverting the logic does not fix all infinite recursion yet either. I don’t know the syntax of what you’re trying to parse, so I cannot easily review the logic for any hints, but maybe you want to review it yourself anyways.

[steffahn]

I cannot explain the debugger’s behavior, though I’m not too familiar with debuggers anyways, so e.g. I couldn’t tell what exactly the “step in” action does. Using println debugging on this code was very straightforward though

[sfackler]

It is very common to see execution jump around a bit in control flow statements in a debugger - the compilation process can result in a sequence of instructions that do not perfectly match a top-down flow from the perspective of the debuginfo.

[J-Cake]

I think this is the most likely explanation. This along with a bad recursive function. Although I'm yet to identify the fault in my function.

@steffahn made mention to parse_index function, which isn't used at all by several examples where this behaviour can be observed, so that may not be the right answer. Regardless, I'll mark this thread as closed, as there isn't really a bug at all.

Thanks again for all the help

[steffahn]

Are you telling me, I'm supposed to do something other than cargo run in the linked repository to reproduce the specific case you're talking about? Because that's all I did, and I assumed it's correct because it resulted in a stack overflow.

If the reproduction is different, you need to explain how to do it

[J-Cake]

nono, that's what I'm doing as well, but there are other inputs that you can use which produces that behaviour without using the parse_index function. That's all

[steffahn]

So maybe other functions besides parse_index also produce infinite recursion? More bugs, and thus more opportunities to improve…? If you tell me how exactly to run one of these other inputs (just modify main or what’s the approach? And what’s the input?), I will gladly do another quick round of printf debugging for you, in case there’s cases left where you feel something weird is going on.

[J-Cake]

I might take you up on that - a fresh set of eyes is always good.

in main.rs, the string in the call to command::parser::tokenise contains the source of the expression being parsed.

There are a few syntax examples in the README.md

fn main() {
  let tokens = command::parse::tokenise("echo('hi')").unwrap();
  let ast = command::parser::parse(&tokens).unwrap();

  println!("{:#?}", ast);
}

[steffahn]

Judging by putting a println statement at the beginning of parse_index and another print statement at each exit point of the parse_index function, all examples in the main function or in the README that I could find would either not overflow the stack, of if they did (the more common case) would infinitely recurse into parse_index calls. Something like this gives that kind of output:

fn parse_index(tokens: &[Token]) -> Result<ASTNode, SyntaxError> {
+   println!("~~ starting parse_index call");
    if tokens.len() != 1 {
+       println!("~~ about to end parse_index call");
        return Err(SyntaxError::InvalidSyntax(tokens[0].line, tokens[0].column));
    }

    let indices = top_level_split(tokens, |t| matches!(t.token_type, TokenType::Dot), false)?;

    let mut indices = indices.into_iter().map(parse);

    if indices.any(|i| i.is_err()) {
+       println!("~~ about to end parse_index call");
        return Err(SyntaxError::InvalidSyntax(tokens[0].line, tokens[0].column));
    }

+   println!("~~ about to end parse_index call");
    Ok(ASTNode::Index(indices.map(|i| i.unwrap()).collect()))
}

and then you can see a log with lots of starting … and (essentially) none of the corresponding about to end … messages in the console.

[J-Cake]

I've put a guard clause at the beginning of the parse_index to catch all expressions without a . in them, which seemed to do the trick.

I wouldn't have found that without you, so thank you