Skip to content

refactor(syntax): generalize the macro repetition separator node - #10310

Open
orizi wants to merge 1 commit into
graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-capturesfrom
graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread
Open

refactor(syntax): generalize the macro repetition separator node#10310
orizi wants to merge 1 commit into
graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-capturesfrom
graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread

Conversation

@orizi

@orizi orizi commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

MacroRepetition.separator was OptionTerminalComma, baking the only separator
the parser currently accepts into the grammar. Replace it with
OptionMacroRepetitionSeparator, wrapping a MacroRepetitionSeparator struct
whose single token child is a TokenNode — the codegen's all-terminals enum —
so the grammar can hold any terminal as a separator.

An option must wrap a node that has a SyntaxKind, and enums do not get one
(generate_kinds_code filters NodeKind::Enum out), hence the small wrapper
struct — the same reason TokenTreeLeaf wraps TokenNode. The syntax crate is
regenerated by generate-syntax, not hand-edited.

Behavior is unchanged. The parser still accepts only TerminalComma in that
slot, and the semantic matcher and expansion still treat only a comma as a
separator, now via a single repetition_separator helper that returns None
for any other token — the same branch as "no separator" — rather than
panicking, so loosening the parser later cannot introduce a panic.
$($x:ident);* is still rejected with the same Missing macro repetition operator diagnostic. The only golden churn is in the parser partial_trees:
OptionTerminalCommaEmpty becomes OptionMacroRepetitionSeparatorEmpty, and
existing comma separators gain the MacroRepetitionSeparator level. No semantic
golden changed.

TokenTreeRepetition.separator is deliberately left as OptionTerminalComma.
The adjacent TODO(Dean): Remove TokenTreeRepetition and TokenTreeParam (it's redundant) intends to delete that node outright, so migrating it would be churn
on code slated for removal and would couple the call-site token-tree grammar to
the macro-rule grammar that TODO wants to disentangle. That refactor is
explicitly not folded in here; .add_option("TerminalComma") stays for it and
becomes dead only once it lands.

The unreachable!() on ast::MacroParamKind::Missing in macro_declaration.rs
is not made reachable by this change, so no guard was added. The accepted token
language is identical before and after, so no new tree shape is produced, and
the separator slot is structurally disjoint from ParamKind. Checked
empirically that it also stays unreachable in general: macro m { ($x:foo) => { 1 }; } reports Missing tokens. Expected a macro rule parameter kind. and the
rule is then dropped by the descendants(..).any(is_missing) filter in
priv_macro_declaration_data, so the matcher never converts a missing param
kind. That filter is unaffected here — OptionMacroRepetitionSeparatorEmpty is
a plain struct kind, not a missing kind, exactly like the
OptionTerminalCommaEmpty it replaces.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

orizi commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from ed9467d to 86cbcf4 Compare August 4, 2026 08:41
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch 2 times, most recently from 931afdd to be641de Compare August 4, 2026 08:56
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 86cbcf4 to 03aed4c Compare August 4, 2026 08:56
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from be641de to e0049dd Compare August 5, 2026 10:54
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 03aed4c to 524388b Compare August 5, 2026 10:54
@orizi
orizi marked this pull request as ready for review August 5, 2026 11:10
@cursor

cursor Bot commented Aug 5, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Structural syntax-tree refactor with behavior-preserving semantic handling; parser still only accepts commas and macro matching/expansion logic is centralized in one helper.

Overview
Replaces MacroRepetition.separator from OptionTerminalComma with OptionMacroRepetitionSeparator, wrapping a MacroRepetitionSeparator node whose token child is a generic TokenNode. The grammar can represent any terminal as a separator; the parser still only accepts a comma today, and semantics route through a new repetition_separator helper that treats non-comma tokens like “no separator” instead of assuming a comma-only shape.

Parser partial-tree goldens pick up the extra wrapper level; observable macro behavior and diagnostics are intended to stay the same. TokenTreeRepetition is unchanged on purpose.

Reviewed by Cursor Bugbot for commit 51dcc85. Bugbot is set up for automated code reviews on this repo. Configure here.

@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from e0049dd to df241a0 Compare August 5, 2026 11:59
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 524388b to 062a85b Compare August 5, 2026 11:59
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from df241a0 to e229982 Compare August 5, 2026 15:15
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 8b2a8e1 to d781f92 Compare August 5, 2026 16:58
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from e229982 to 6a96549 Compare August 5, 2026 16:58
@orizi
orizi changed the base branch from graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures to graphite-base/10310 August 5, 2026 17:15
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from 6a96549 to adb52a4 Compare August 6, 2026 11:33
@orizi
orizi force-pushed the graphite-base/10310 branch from d781f92 to 9fd1f92 Compare August 6, 2026 11:33
@orizi
orizi changed the base branch from graphite-base/10310 to graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures August 6, 2026 11:34
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 9fd1f92 to 648a842 Compare August 12, 2026 20:54
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from adb52a4 to 8c0b736 Compare August 12, 2026 20:54

@eytan-starkware eytan-starkware left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eytan-starkware+AGNT made 2 comments.
Reviewable status: 0 of 8 files reviewed, 1 unresolved discussion (waiting on eytan-starkware, orizi, and TomerStarkware).


a discussion (no related file):
Note: the comments below are from an automatic orizi-review run (Claude agents reviewing in Ori's style, findings adversarially verified before posting). Treat with the usual bot skepticism.


crates/cairo-lang-syntax-codegen/src/cairo_spec.rs line 911 at r1 (raw file):

    )
    .add_option("MacroRepetitionSeparator")
    .add_option("TerminalComma")

after this change OptionTerminalComma exists only for TokenTreeRepetition (l.825). move it up next to that node so it's obvious it dies together with the TODO(Dean): Remove TokenTreeRepetition.

`MacroRepetition.separator` was `OptionTerminalComma`, baking the only separator
the parser currently accepts into the grammar. Replace it with
`OptionMacroRepetitionSeparator`, wrapping a `MacroRepetitionSeparator` struct
whose single `token` child is a `TokenNode` — the codegen's all-terminals enum —
so the grammar can hold any terminal as a separator.

An option must wrap a node that has a `SyntaxKind`, and enums do not get one
(`generate_kinds_code` filters `NodeKind::Enum` out), hence the small wrapper
struct — the same reason `TokenTreeLeaf` wraps `TokenNode`. The syntax crate is
regenerated by `generate-syntax`, not hand-edited.

Behavior is unchanged. The parser still accepts only `TerminalComma` in that
slot, and the semantic matcher and expansion still treat only a comma as a
separator, now via a single `repetition_separator` helper that returns `None`
for any other token — the same branch as "no separator" — rather than
panicking, so loosening the parser later cannot introduce a panic.
`$($x:ident);*` is still rejected with the same `Missing macro repetition
operator` diagnostic. The only golden churn is in the parser partial_trees:
`OptionTerminalCommaEmpty` becomes `OptionMacroRepetitionSeparatorEmpty`, and
existing comma separators gain the `MacroRepetitionSeparator` level. No semantic
golden changed.

`TokenTreeRepetition.separator` is deliberately left as `OptionTerminalComma`.
The adjacent `TODO(Dean): Remove TokenTreeRepetition and TokenTreeParam (it's
redundant)` intends to delete that node outright, so migrating it would be churn
on code slated for removal and would couple the call-site token-tree grammar to
the macro-rule grammar that TODO wants to disentangle. That refactor is
explicitly not folded in here; `.add_option("TerminalComma")` stays for it and
becomes dead only once it lands.

The `unreachable!()` on `ast::MacroParamKind::Missing` in `macro_declaration.rs`
is not made reachable by this change, so no guard was added. The accepted token
language is identical before and after, so no new tree shape is produced, and
the separator slot is structurally disjoint from `ParamKind`. Checked
empirically that it also stays unreachable in general: `macro m { ($x:foo) => {
1 }; }` reports `Missing tokens. Expected a macro rule parameter kind.` and the
rule is then dropped by the `descendants(..).any(is_missing)` filter in
`priv_macro_declaration_data`, so the matcher never converts a missing param
kind. That filter is unaffected here — `OptionMacroRepetitionSeparatorEmpty` is
a plain struct kind, not a missing kind, exactly like the
`OptionTerminalCommaEmpty` it replaces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/capture-fidelity.f1-parenthesize-expr-captures branch from 648a842 to d42096c Compare August 16, 2026 11:37
@orizi
orizi force-pushed the graph-plan/2026-08-03-macro-fixes/separator-grammar.sep-grammar-thread branch from 8c0b736 to 51dcc85 Compare August 16, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants