fix: scanner-assisted mid-command redirect detection

[AMZN-hgoffin]

Problem

Redirects between arguments (e.g. grep 2>/dev/null -q pattern file) produce incorrect parse trees. The file_redirect rule uses repeat1($._literal) for its destination, so it greedily consumes all subsequent words as redirect destinations instead of leaving them as command arguments.

Approach

The external scanner peeks ahead past redirect operators and their destinations. If more non-redirect words follow, the scanner emits a mid-command token that keeps the redirect inside command's repeat. If no words follow, the redirect is trailing and uses redirected_statement as before.

Two new grammar rules (_mid_command_file_redirect and _mid_command_plain_redirect) are aliased to file_redirect, so node-types.json is unchanged. Consumers that already handle redirect fields on command nodes -- which is required for pre-command redirects like 2>&1 cmd -- need no changes.

What's fixed

All redirect operators as mid-command: > >> < 2> 2>&1 >& <& >| &> &>> >&- <&-. Chained redirects (cmd >a >b arg), process substitution destinations (cmd > >(tee log) arg), herestrings (cmd <<< "input" arg), and close-fd operators (cmd >&- arg). Also adds redirect support inside [ ] test brackets ([ -f file 2>/dev/null ]).

Compatibility

Trailing redirects still produce redirected_statement exactly as before. The one known tree shape change is for trailing redirects inside backtick command substitutions (e.g. echo `cmd >file` arg), where the redirect moves from redirected_statement to a direct child of command. Backticks use the same symbol to open and close, making it impractical for the scanner's lookahead to determine whether a backtick terminates the current context. This is safe for consumers because command already accepts redirect children for the pre-name case (e.g. 2>&1 cmd), so all consumers must already handle redirects on command nodes.

Known limitations (pre-existing, same as master)

• Redirects between -a/-o in test expressions: [ -f file 2>/dev/null -a -d dir ]
• Pre-expression fd-prefixed redirects in test brackets: [ 2>/dev/null -f file ]

Both are rare in practice. They are legal because [ is a command like any other and the shell strips redirects before [ sees its arguments. Interior redirects are not legal on [[ ]] constructs, only [ ].

Related

• Incorrect parse of arguments after a redirect #233 -- original bug report identifying repeat1 as root cause
• fix: redirect should consume exactly one destination #331 -- alternative fix that changes file_redirect to single destination and adds argument fields to redirected_statement. That is a simpler grammar change but a breaking change to node-types.json requiring consumers to handle the new argument field. This PR preserves the existing tree structure for trailing redirects and only changes behavior for inputs that previously produced incorrect trees.

[AMZN-hgoffin]

Maintaining compatibility with the existing parse shapes did require some ... rather intense scanner hackery.

I'm not upset if this PR is not taken, I just want it to be an option because right now the parse deviates badly from bash's actual behavior with mid-command and mid-test redirects. PR #331 is a much simpler fix that gets you about 80% of the way there.

Once I cracked the scanner open, I couldn't stop myself from dealing with various incorrect parses of "fd-close" and other redirection bugs that I was going to have to touch on anyway. I did hit a lot of birds with this stone. But, it is a very ugly and complex stone. Happy to remove some of the inline scanner comments if you think they break up the code too much (eg, the enum definitions).

If retaining tree shape isn't a concern and we are willing to just force consumers to accept interleaved redirects on all commands and test_commands, I think the line count of the patch can be cut roughly in half. But that also feels like a breaking change that might hurt consumers of this project.