[yaml, yaml_edit] Add token stream to package:yaml and rewrite yaml_edit using token-driven CST

[sigurdm]

Adds optional token stream retention to package:yaml and rewrites package:yaml_edit's modification logic around a lossless token-driven Concrete Syntax Tree (CST) and slot-directed mutations.

Motivation

package:yaml produces a value tree: it describes what a document means, but not how it is written. Comments, blank lines, indentation, delimiter positions, and collection styles are absent from the AST. Previously, YamlEditor recovered this information during edits by scanning the source string with ad-hoc heuristics (indexOf, lastIndexOf, regex matching) to locate colons, commas, brackets, and indentation levels. This made edge cases (comments containing delimiter characters, multiline comments, nested flow collections, and mixed indentation) prone to offset miscalculations and syntax corruption.

This PR replaces ad-hoc scanning with a tiling Concrete Syntax Tree (CST) constructed once from package:yaml's AST and token stream:

• Exact Tiling Invariant: Every character of the source belongs to exactly one CST slot, slots are contiguous and disjoint, and concatenating all slots reproduces the source byte for byte. CstDocument.parse enforces this invariant on every document it builds, verifying that every structural indicator matches its expected character and every interstitial gap contains only whitespace and comments.
• Structural Locality: Because slots are disjoint, replacing the source range of any slot is guaranteed to leave every other character of the document untouched. Mutations never search the source string for delimiters—every offset needed for an edit is already a verified slot boundary.

Changes

package:yaml (3.2.0-wip)

• Optional Token Stream (retainTokens: true):
  • Adds retainTokens parameter to loadYamlDocument to retain scanner tokens on YamlDocument.tokens.
  • Exports Token, TokenType, ScalarToken, CommentToken, TagToken, AnchorToken, AliasToken, and directive token classes via package:yaml/tokens.dart.
  • Emits CommentToken for all comments and ensures YamlDocument.tokens contains strictly positive-length, non-overlapping real tokens sorted in source order.
  • Updates keep-chomped (|+, >+) block scalar token spans to include trailing empty lines consumed by the scalar.

package:yaml_edit

• Lossless Token-Driven CST (lib/src/cst.dart):
  • Builds a concrete syntax tree directly from package:yaml's token stream (loadYamlDocument(..., retainTokens: true)), avoiding custom character scanning in yaml_edit.
  • Represents every token, delimiter, comment, newline, and whitespace segment as a CstNode with exact character spans.
  • Includes an invariant verifier (_checkTiling) ensuring the CST tiles the input string with zero gaps and zero overlaps.
• Slot-Directed Mutations (lib/src/cst_mutations.dart):
  • Replaces heuristic string searching in list_mutations.dart and map_mutations.dart with syntax-directed CST operations.
  • Mapping insertions and updates target exact key/colon/value slots and compute indentation from sibling key columns.
  • Flow collection mutations locate enclosing delimiters ({...}, [...]) and comma tokens directly from the CST while preserving leading and sibling trailing comments.
  • Block sequence additions and removals operate directly on dash tokens and entry spans.
• Testing:
  • Adds CST tiling verification across yaml-test-suite (test/cst_tiling_test.dart).
  • Adds structured property-based CST fuzzing (test/structured_fuzz_test.dart) verifying CST tiling, structural locality frame bounds, comment conservation laws, algebraic lens laws (PutGet, GetPut, PutPut), insert-remove cancellation, and byte-for-byte disjoint path commutativity.

[gemini-code-assist]

Code Review

This pull request introduces a Concrete Syntax Tree (CST) layout engine to represent non-semantic source elements like comments and whitespace, allowing the editor to support updating explicit keys without colons and resolve previous crash test failures. Feedback focuses on improving the robustness of finding list item hyphens by scanning forward rather than using lastIndexOf, and optimizing performance in map mutations by avoiding redundant .toList() conversions on map keys.

[gemini-code-assist]

Using yaml.lastIndexOf to find the hyphen offset can incorrectly match a hyphen inside a comment or a string value that precedes the current element. Since we parse sequentially, we can scan forward from prevLineEnd to trueSpan.start.offset while skipping comments and whitespace to reliably find the actual list item hyphen.

      var hyphenOffset = -1;
      if (list.style == CollectionStyle.BLOCK) {
        var scan = prevLineEnd;
        while (scan < trueSpan.start.offset) {
          final c = yaml.codeUnitAt(scan);
          if (YamlChar.isWhitespace(c) || YamlChar.isLineBreak(c)) {
            scan++;
          } else if (c == YamlChar.hash) {
            scan++;
            while (scan < yaml.length &&
                !YamlChar.isLineBreak(yaml.codeUnitAt(scan))) {
              scan++;
            }
          } else if (c == 0x2D /* - */) {
            hyphenOffset = scan;
            break;
          } else {
            scan++;
          }
        }
      }

[sigurdm]

Resolved: replaced backwards scan with forward scan from prevLineEnd that skips comments and whitespace to locate the block sequence hyphen indicator.

[gemini-code-assist]

Calling map.nodes.keys.toList() repeatedly inside this block creates redundant list copies. We can optimize this by converting the keys to a list once and reusing it.

Suggested change

	final keyAtIndex = map.nodes.keys.toList()[insertionIndex] as YamlNode;
	final keySpanStart = keyAtIndex.span.start.offset;
	final prevNewLineIndex = yaml.lastIndexOf('\n', keySpanStart);
	final int minOffset;
	if (insertionIndex > 0) {
	final prevKey = map.nodes.keys.toList()[insertionIndex - 1];
	final keys = map.nodes.keys.toList();
	final keyAtIndex = keys[insertionIndex] as YamlNode;
	final keySpanStart = keyAtIndex.span.start.offset;
	final int minOffset;
	if (insertionIndex > 0) {
	final prevKey = keys[insertionIndex - 1];

[sigurdm]

Resolved: cached keys list locally and used Iterable.elementAt in _addToFlowMap to avoid redundant allocations. Also rewound scan past leading indentation spaces so entry insertion at index 0 preserves sibling column indentation.

[jonasfj]

This would effectively introduce a new YAML parser, correct?

Might it not be wiser to add additional metadata to the existing YamlNode objects?

just an idea, ofcourse we'd want to avoid extra cost for parsing YAML for people who don't care.

[sigurdm]

Yeah - I think we basically need to bring this all the way to a full parser... maybe we extend package:yaml we should try both directions to see what the costs are....