fix: preserve common abbreviation continuations - #155
mldangelo-oai merged 37 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (1)ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99614ba7e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 106f403ffe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3064f92f53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
🔵 Needs a closer look
Four unresolved moderate findings in src/utils.ts need to be addressed.
Pull request overview
Fixes sentence segmentation for vs. continuations and U.S. geographic phrases.
Changes:
- Adds
vsto abbreviation tables. - Updates handling for
U.S. SenateandU.S. Commission. - Adds segmentation and ROUGE-L regression tests.
File summaries
| File | Summary |
|---|---|
test/tests.ts |
Adds regression coverage for abbreviation and geographic phrase handling. |
src/utils.ts |
Updates boundary logic. Four unresolved moderate findings (1 vote each) affect paragraph preservation, unspaced U.S. splits, vs. proper-name handling, and geographic continuation logic. |
src/constants.ts |
Registers vs abbreviations. |
Review details
Suppressed comments (4)
src/utils.ts:367
- The paragraph exception is not reached when the wrap contains two line terminators:
abbreviationstill ends in\n/\r\n, soabbrvReg.test(abbreviation)is false and execution falls through to thestartsWithTitleCasemerge branch. As a result, inputs such as the newU.S.\n\nSenatecases are joined instead of preserving the paragraph boundary. Trim all whitespace before the abbreviation match while retaining the originalsuffixfor the paragraph check.
!/[\r\n][^\S\r\n]*[\r\n]/.test(suffix.replace(/\r\n/g, '\n'))
src/utils.ts:799
- The new unspaced
U.S.Senate/U.S.Commissioncases still cannot split at theU.S.period.isUnspacedSentenceBoundaryreaches this branch withisAbbreviationExceptionfalse, but the laterdottedIdentifiercondition matches theS.suffix followed by the capitalized continuation and suppresses the boundary. Exclude geographic acronyms from that dotted-identifier suppression so the expected unspaced-boundary assertions can pass.
abbrvReg.test(gateSuffix) && isAbbreviationException(gateSuffix, following);
src/utils.ts:141
- This broad independent-sentence check misclassifies proper-name continuations after
vs.. For example,The Giants vs. Boston Celtics, which was televised.matchesBoston Celtics,and therefore splits atvs., even though the comma-terminated phrase is the opponent name. Please narrow the terminal-sentence heuristic or add a proper-name guard before treating this continuation as independent.
independentSentenceReg.test(continuation) ||
/^(?:this|that|these|those|it|we|they|he|she|i)\b/i.test(continuation))
src/utils.ts:407
- Adding
senateandcommissionto this shared continuation regex also affects the normal abbreviation-merge branch below. ForU.S.Senate/U.S.Commission,sentenceChunkscreates a boundary atU.S., butnextChunkhas no separating whitespace and this condition still classifies it as a geographic continuation, so theelsebranch appends it and defeats the new unspaced-boundary expectations. Restrict this guard to whitespace-separated chunks (or use a separate predicate) so those boundaries remain splits.
!isAbbreviationException(gateSuffix, nextChunk) &&
!(
geographicAcronymReg.test(gateSuffix) &&
geographicContinuationReg.test(nextChunk.trim())
)
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Reviewed all four suppressed Copilot suggestions. Fixed the genuine opponent-name split in The other three suggestions do not reproduce: CR/LF paragraph boundaries and |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0af2de6b6c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved moderate issues remain around quoted continuations and paragraph-boundary detection.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
src/utils.ts:367
- These geographic continuation checks are bypassed for the normal line-wrap shape.
sentenceChunksemits the separator (\n,\n\n, etc.) as its own chunk, so the pending buffer reaches this branch withgateSuffixending in the newline;trimEnd()only removes spaces, makingabbrvReg.test(abbreviation)false. The fallbackchunk.startsWithTitleCasethen merges bothU.S.\nSenateandU.S.\n\nSenate, collapsing the paragraph boundary (and similarly the quoted case). Apply the blank-line-aware distinction on the pending-separator path before merging.
!(
geographicAcronymReg.test(abbreviation) &&
geographicContinuationReg.test(nextChunk?.trimStart() ?? '') &&
!/[\r\n][^\S\r\n]*[\r\n]/.test(suffix.replace(/\r\n/g, '\n'))
)
src/utils.ts:356
vsis now treated as an exception for every title-cased continuation, but the line-wrap fallback does not distinguish a paragraph break. ForThe Giants vs.\n\nBoston Celtics, which was televised.,isAbbreviationExceptionmakes the first branch false and thestartsWithTitleCasefallback joins the chunks, losing the blank-line boundary. The paragraph check added for geographic continuations should also cover thisvs.path (or the fallback should refuse to merge across a paragraph).
!isAbbreviationException(abbreviation, nextSentence) &&
src/utils.ts:366
- The paragraph test only works when both line terminators fit in
SentenceBuffer.suffix, but that suffix is capped atsentenceSuffixLength(10 here). A formatted blank line such asU.S.\n \nSenate voted.truncates the first newline from the suffix, so this regex sees no paragraph and the new geographic continuation is merged into the preceding sentence. Detect the paragraph separator from the full buffered text (or carry that fact separately) rather than from the bounded suffix.
!/[\r\n][^\S\r\n]*[\r\n]/.test(suffix.replace(/\r\n/g, '\n'))
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
🔵 Needs a closer look
Four unresolved moderate findings remain in src/utils.ts, involving repeated scans and smart-quote boundary handling.
Review details
Suppressed comments (4)
src/utils.ts:687
- This still becomes quadratic for the repeated leading-elision case: after the closer is cached, every candidate calls
hasLaterAngleElisionOpening(input, start, found)and rescans all later openers. The 50,000-candidate regression input intest/tests.ts:3688therefore repeatedly traverses the same suffix and can exceed its 3-second limit; cache the next valid elision opener or advance a shared cursor instead of scanning from eachstart(the claimed linear/monotone scan is not achieved here).
return hasLaterAngleElisionOpening(input, start, found) ? -1 : found + closer.length;
src/utils.ts:1008
closingDelimiterEndonly consumes ASCII/Treebank closers, even thoughpairedBracketQuoteEndnow recognizes the smart-quote endpoints. For example,He wrote “vs.” Alice explained.stops at the period before”; the next character is neither whitespace nor a cased character, so the quoted-versus boundary is lost and the whole input remains one sentence. Consume the validated smart closer (including the spaced-closer path) and haveclosingDelimiterContexttreat that endpoint as a completed quotation.
closingDelimiterReg.test(input[end]) &&
!isUnmatchedAngleCloser(input, end, quotePending, brackets)
) {
src/utils.ts:369
- The new versus continuation path strips only ASCII quote/bracket openers from
nextChunk. With a supported smart opener,The standard abbreviation is vs. “This is clearer.”leaves“at the start, so bothstartsWithCasedCharacterandstrIsTitleCasefail and the terminalvs.is merged with the next sentence. Normalize the same supported opening-quote set here and in the non-line-break path at line 424.
const nextSentence = nextChunk?.replace(/^[\s"'([{<]+/, '');
src/utils.ts:1174
- The unspaced-delimited shortcut still recognizes only ASCII opening quotes. Consequently
Use vs.“Alice replied.”cannot take this path:sentenceEndsees the non-whitespace“after the period and rejects the boundary, so the two sentences stay joined. Share the supported quote-opener predicate (including the smart/Treebank forms recognized above) instead of hard-coding this narrower character class.
if (
insideQuotes ||
index < brackets.bracketQuoteEnd ||
brackets.depth > 0 ||
!/["'([{<]/.test(input[next] ?? '')
) {
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eff69bcd83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Individual dispositions of Copilot review PRR_kwDOMlO9es8AAAABNj7siQ at unchanged eff69bc:
The final source remains the independently reviewed952-test tree. These dispositions preserve the declared boundary grammar and correct the inaccurate performance/parent-output claims. |
Read the full marked initial with its complete Unicode predecessor while retaining the existing identifier character categories and A/I boundary rule. Cover circled and enclosed bases, long marks, and predecessor counterexamples.
|
@codex review Please review the current integration at |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75e13a2a6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🔵 Needs a closer look
Three moderate issues and one nit remain unresolved in src/utils.ts.
Review details
Suppressed comments (4)
src/utils.ts:423
- A paragraph separator is appended to
pendingand then reaches this branch, wherestartsWithTitleCasemerges it regardless ofcontinuesAbbreviation. Consequently inputs covered by the new paragraph tests, such asThe Giants vs.\n\nBoston Celtics won.andHe said Linux vs.\n\nWindows today., are joined instead of retaining the paragraph boundary (the same happens for both casing modes). Gate the title-case line-wrap merge on the absence of a paragraph break, while carrying the recognized outer-quote state so paragraph breaks inside paired quotes still merge.
(chunk.startsWithTitleCase ||
continuesAbbreviation ||
src/utils.ts:397
- The paragraph test is performed on
suffix, butSentenceBuffer.suffixis intentionally capped atsentenceSuffixLength(8 here). A separator such as the new test case\n \nis longer than that window, so the first newline is discarded and this expression sees only one newline; the versus abbreviation is then treated as a single line wrap and merged instead of splitting. Detect paragraph breaks from the full separator chunk (or preserve separator metadata) rather than from the bounded suffix.
const separator =
suffix.slice(suffix.trimEnd().length) + (nextChunk?.match(/^\s*/)?.[0] ?? '');
const continuesAbbreviation =
!/\n[^\S\n]*\n/.test(separator.replace(/\r\n?/g, '\n')) &&
src/utils.ts:2003
- The
index < brackets.bracketQuoteEndguard suppresses the early unspaced-boundary check when a terminal is followed by its recognized plain-single closing quote.sentenceEndthen consumes that quote, butisUnspacedSentenceBoundarysees the following(/[/{asnextand rejects it because it only accepts a letter or number, so cases such asHe wrote 'vs.'(Alice replied.)remain joined instead of producing the two sentences asserted by the new tests. The boundary logic needs to account for a completed quote before an unspaced opening delimiter.
insideQuotes ||
index < brackets.bracketQuoteEnd ||
brackets.depth > 0 ||
!/["'([{<]/.test(input[next] ?? '')
src/utils.ts:1492
- The mask is allocated at the first
<, before this scan knows whether any unquoted pair exists, and is then discarded whenpairs === 0. Large inputs containing only unmatched or quoted angle marks therefore incur an unnecessaryinput.lengthallocation (potentially doubling peak memory); defer creating the mask until the first confirmed>or track pending opener positions until a pair is found.
if (matched.length === 0) {
matched = new Uint8Array(input.length);
}
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
Critical dispositions of all four suppressed notes in Copilot review PRR_kwDOMlO9es8AAAABNnq2vg, checked against 75e13a2 and the exact 9d97d7b source:
The first three dispositions include 36 focused inputs and 144 actual API calls across the published and fixed source. All 1,343 tests and local checks pass. COMMENTED bot reviews are not formal approvals. Final-head code/security/Copilot reviews and CI remain required. |
|
@codex review Please review code and security on exact head 9d97d7b. The three-site single-guillemet ownership repair and its ten regression cases are the new change; all 1,343 local tests pass. Both prior Codex findings and all four suppressed Copilot claims have explicit evidence-based dispositions above. Review those critically and report any remaining actionable issue. |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
🔵 Needs a closer look
Four moderate review findings remain unresolved in src/utils.ts.
Review details
Suppressed comments (4)
src/utils.ts:423
- When a comparison
vs.is followed by a blank paragraph,sentenceChunksdoes not emit a boundary (the comparison exception keeps the period pending), so the prefix and the following sentence arrive here as one line-wrappedchunk. Becausechunk.startsWithTitleCaseis true for inputs such asThe Giants vs.\n\nBoston Celtics..., this branch joins across the paragraph and bypasses the paragraph guard above, contrary to the new paragraph-boundary cases. The title-case fallback needs to distinguish an enclosing recognized quote/bracket (which must keep embedded comparisons together, including Treebank quotes) from an unquoted paragraph before joining.
(chunk.startsWithTitleCase ||
continuesAbbreviation ||
src/utils.ts:1945
- When a completed quotation makes
delimitedVersustrue, this admits numeric sentence starts, but the quantity guard only recognizes a unit when it is immediately followed by sentence punctuation/end. ConsequentlyHe wrote "vs." 100 times correctly.is treated as two sentences instead of staying attached, contradicting the new expectation attest/tests.ts:2591; extend the supported quantity-fragment check to cover the trailing words while keeping cases such as2 days passed.split.
(!abbrvReg.test(suffix) || delimitedVersus) &&
!/^\S+(?:\s*%|\s+(?:time|year)s?\b|\s+(?:month|week|day|hour|minute|second|star|point|percent)s?(?=\s*[.!?](?:\s|$)|\s*$))/iu.test(
input.slice(next),
)
src/utils.ts:423
- The new continuation path appends a wrapped abbreviation chunk to
pendingwhile the buffer still contains the CR/LF. On the next iteration there is no following chunk, so thehasLineBreaksfallback splitschunk.text()back into separate lines; lowercase comparisons and geographic continuations such asandroid vs.\nWindowsand2026 U.S.\nSenatetherefore cannot normalize to one sentence as the added tests require. Normalize the buffer before storing it inpending(or otherwise mark the joined wrap as consumed).
(chunk.startsWithTitleCase ||
continuesAbbreviation ||
src/utils.ts:1406
- This later-opener guard is stricter than
angleQuoteCloser: it rejects an ASCII quote preceded by punctuation such as:or,, even thoughangleQuoteCloserclassifies that position as a valid opener. A leading elision such as'Tiscan therefore borrow that later quote as its closer, causing the scan to skip an intervening<...>span and leave its>unmatched. Use the same opener predicate here so punctuation-adjacent independent quotations are protected too.
angleQuoteCloser(input, next) !== undefined &&
(opener === '‘' || /^[\s([{<]$/.test(input[previous])) &&
/\S/.test(input[next + 1] ?? '')
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
Critical dispositions of all four suppressed notes in final-head Copilot review PRR_kwDOMlO9es8AAAABNn9LNQ, verified on
The bounded comparison made 616 actual API calls over 77 inputs on current main, original #155, this head and the isolated proposed predicate. All 154 case-mode outputs on this head match original #155 for these probes. The initial failed assumption about fragment joining is preserved in the local evidence; it is not counted as a passing test. All 1,343 regression tests and all ten actual GitHub checks pass. Node 18/20/22/24 CI each checked out the exact main/head merge and passed installation, types, strict lint, build, packed consumers and tests. Codex code/security reviews are clean. These are critically reconciled COMMENTED reviews, not formal approving reviews; the user's authorized approval override is separate. |
Summary
Keeps standard versus comparisons and U.S. Senate/Commission phrases together while preserving supported terminal-versus boundaries. Both existing spellings,
vs.andv.s., share the same rules, including numeric and lowercase line wraps. Question/exclamation marks and explicit paragraphs outside open delimiters remain sentence boundaries.Terminal versus inside completed ASCII/Treebank quotations releases a following letter- or number-starting sentence; existing quantity fragments stay attached. Complete standalone bracket clauses can end a sentence. Quoted labels respect pending enclosing quotations and brackets. Embedded main-sentence comparisons such as
He noted (Linux vs.\n\nWindows) today.remain attached across paragraphs. Bracket state ignores quoted literal brackets in double, Treebank, paired plain-single, and existing smart quotation forms while separate omission-marker evidence preserves quoted[...]behavior.The broad comma-clause guess was removed. Versus-specific continuation checks use existing copula/pronoun evidence, and other abbreviations retain their prior continuation inputs and Unicode case-folded evidence. Genuine terminal-versus statements clear provisional bracket context; other bare abbreviations retain enclosing context. Spaced Treebank closers, escaped angle quotations, and embedded parentheses across physical line wraps are covered.
Angle matching uses a byte mask and scalar depths, with linear scans and no nesting cap. Quotation scans reuse validated closer positions and monotone cursors, skip matched spans, respect odd/even escape parity, and prevent bounded leading elisions from borrowing a closer across a later independent quotation.
Bracket tracking reuses the existing matched-quote recognizer and monotone cursor, retaining contraction, possessive, and leading-elision controls without changing general quotation-boundary policy.
An inner single or double quote cannot release a pending recognized outer quotation; fully consumed outer and Treebank closers retain boundary behavior. Cached escape validation prevents repeated backslash-run scans across leading-elision candidates. The bounded leading-elision forms include tis/twas, numeric prefixes, cause/em/til/till; genuinely paired quotations with those spellings remain recognized.
Confirmed unspaced bracket boundaries retain their identity through abbreviation handling. Wrapped final embedded clauses normalize with their prefix, and unmatched unquoted
>markers stay with following text. Quoted literal markers retain their existing behavior. Later-opener checks use the same escape-aware quote recognizer as the angle scan.Paired closing quotation marks remain attached when followed immediately by an opening bracket; the unspaced boundary shortcut reuses the existing paired closing offset.
Isolated paired single backticks protect literal angle operators; doubled Treebank quotes keep precedence. Provisional versus context uses the same leading-delimiter normalization as the existing pronoun continuation rule.
Confirmed paired quotation endpoints protect escaped literal brackets and keep inner ASCII/Treebank closers from releasing a pending outer span. Sparse source chunk metadata prevents the versus paragraph override from splitting a still-pending paired quotation. Ordinary copular periods can still divide a multi-sentence quotation; only consuming an inner closer requires the outer endpoint to be complete.
The pronoun continuation rule requires the existing pronoun to end before any Unicode identifier or dash continuation. Hyphenated and accented opponent names stay attached, while genuine pronouns and apostrophe contractions retain their supported boundary.
Documented ambiguities
remains vs. Tomorrow...andvs.Tigersretain comparison precedence; a broad following-clause rule would split validAndroid vs. Windows is common.comparisons.The abbreviation is vs.can end a sentence within a longer quotation, including a missing final quote. This preserves the original vs. behavior; v.s. standardization intentionally adopts that preference. A complete quotation may contain multiple sentences.IT/itpronoun ambiguity remains. Copularis v.s. rather than versusretains the terminal preference in neutral mode.2 days laterand2 days passedshares the numeric-sentence preference.U.S. (He wrote vs.) Alice replied.remains one span, consistently with v.s. and protected e.g./Co. parentheticals. This is an introduced conservative attachment tradeoff for vs.; the claimed intermediate U.S. split did not reproduce.The Giants vs.\n# Results are below.; this PR does not add Markdown parsing.(android vs.\n\nwindows)retains the base paragraph split, unlike an embedded clause in a main sentence.Co.followed by a physical line break retains its prior boundary preference.Angle matching uses lexical paired delimiters. The established Treebank pair and isolated single-backtick cue do not constitute a Markdown or HTML parser; arbitrary backtick-run lengths and HTML element contents retain that ambiguity. Exact multi-backtick and HTML cross-sentence examples keep their baseline output.
Word-final apostrophes such as rockin' can be elisions or real quotation closers. The introduced quoted-versus policy retains closing-quote priority, as required by actual quoted words followed by independent versus sentences; no growing word-final elision vocabulary guesses the intended reading.
Paired single-backtick literals reuse the existing quote endpoint cache for all surrounding bracket types, so literal bracket marks cannot prematurely release a quoted versus boundary. Unpaired backticks remain ordinary punctuation under the existing rule. The original unmatched compact-angle comparison before a blank paragraph still follows the conservative buffer delimiter policy; this baseline limitation is separate from matched-angle recognition.
Backslash parity keeps escaped bracket marks literal in the source bracket state and matched-angle prepass. German low-double/low-single quotations share the existing endpoint cache in both scans; complete inner English pairs retain their role before a compatible German closer. The existing reverse angle-mask phase is unchanged. Quotation-role ambiguity still uses bounded structural preferences.
Second-person You retains the original comparison preference: both
Me vs. You won the match.and a possible new sentence beginning You remain joined without stronger evidence. The bounded supported pronoun rule is not a full subject parser; no new vocabulary is added for this baseline ambiguity.Unquoted copular-versus boundaries retain the original cased-start gate; numeric starts such as
The abbreviation is vs. 2026 brought changes.remain joined. Standalone smart-quote terminals and smart-quoted next starts retain the original general boundary policy, including the unspaced form. Smart paired endpoint recognition in this change protects enclosing quotation/bracket context and does not add a general smart-quote terminal parser.Single guillemets (
‹…›) now share the existing quote endpoint cache in both bracket and angle scans. Literal brackets inside these quotations stay quoted; real surrounding closers and escape parity retain their existing roles.An unterminated tail after a physical wrap, such as
android vs.\nWindowsor2026 U.S.\nSenate, retains the inherited two-fragment result. The supported completed-sentence wraps normalize to one span; this change does not add a general fragment parser.Integration and validation
This head incorporates the landed #156/#169/#173 fixes. The parser preserves list-marker ownership, full Unicode initial context, whole-summary case folding, and both ROUGE-L work limits. A terminal guard uses an early continue to keep the combined scanner within the existing complexity limit.