Fix some lexer rules - #2330
Merged
Merged
Conversation
String continuation includes all of the following TAB/LF/CR/SP characters immediately following the next line. This was clarified and documented in rust-lang#1042, but we didn't update the grammar at the same time. This corrects that to include those spaces. This fixes an issue with strings such as: $' "string\\\n\n\r\tcontinuation"' (using shell escaping to illustrate newlines and carriage returns). There the bare CR should be allowed due to this special-casing for continuations.
This fixes an issue where OUTER_BLOCK_DOC was allowing bare CR as the first character when it shouldn't have. This fixes an issue with comments such as: $'/**\r CR starting block doc comment */' (using shell escaping to illustrate newlines and carriage returns). There the bare CR should not be allowed and should result in a parse error.
This fixes an issue with the BLOCK_COMMENT grammar as it is used within the SHEBANG rule. The problem is that BLOCK_COMMENT was changed in rust-lang#2191 so that it relied on the order of how the comments were handled in the COMMENT rule to avoid ambiguity with OUTER_BLOCK_DOC. However, because SHEBANG is using the BLOCK_COMMENT rule directly, we need BLOCK_COMMENT to be able to stand on its own without relying on the order in COMMENT. This fixes an issue with shebang such as: #! /** doc */ [attr] Here this should be treated as a shebang (because it is a doc block comment). The solution is to do the appropriate negative lookahead to exclude OUTER_BLOCK_DOC and INNER_BLOCK_DOC.
rustbot
added
the
S-waiting-on-review
traviscross
approved these changes
Aug 18, 2026
traviscross
enabled auto-merge
August 18, 2026 22:46
traviscross
removed
the
S-waiting-on-review
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes some issues with string continuations, comments, and shebang identified by @mattheww in #2325 (comment).
The SHEBANG grammar still has problems as identified in #2325 (comment) that are more difficult to fix. In particular, the cut operator in BLOCK_COMMENT is causing a parse error in the negative lookahead, but rustc ignores those. This might require more significant changes than I have time for at the moment.