Logos Parsing Fails only on Windows

This post is also published as an issue on the original repo. The content is similar to the related issue.


I have a strange issue where logos fails tests only on Windows builds.

This is how I define the token types. I test it across this custom script.

While MacOS and Linux builds pass the tests, Windows build oddly fails while parsing a token.

Troubleshooting

Windows Paths

At first, I have thought maybe it is related Windows paths when I include_str here since it might require backwards slashes. However, it is not the case, the file is read successfully.

Windows Whitespace Encoding

This is the only thing I am suspicious about. I define there might be a whitespace at this line as [ ]. Maybe the encoding in Windows is not the same. What do you say?


Environment

  • Rust 1.43.1
  • logos 0.11.4

GHA's checkout script checks out on Windows with a core.crlf setting of "check out CRLF, commit LF". Given I don't see a handling of \r, I suspect that's your issue. While actual Rust code, including string literals, is normalized to have LF line endings, include_str! includes the file literally, and does not normalize CRLF to LF.

Use #[regex(r"[ \t\r\n\f]+", logos::skip)], I suspect that will fix the issue. (Or normalize CRLF before hitting logos, that's another option.)

1 Like

Yeah, that was the problem. Thanks for giving insight.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.