refactor: separate text preprocessing from numeric utilities - #172
Conversation
…/codex/question-annotation-boundaries-20260915
Preserve current citation and ellipsis ownership while retaining the reviewed quotation, modifier-footnote and scoped-question behavior. Keep contributor history and the complete combined regression suite, including the original bounded-memory fixtures and explicit policy adaptations for later parser fixes.
Integrate the exact landed main commit after validating the combined parser. Preserve all validated tracked-file bytes.
|
@codex review Please review code and security on exact final integrated head 8723c42, tree 00a180013eafa04354ef86472ce712ce9c643b44, against current main 2d2a628. This is a pure extraction of the now-landed #170 source into sentence-segment, tokenize and shared text modules, with the existing utility re-exports and an AGENTS source-map update. Independent proofs preserve all 305 declaration bodies, 1,179 nested bodies, 187 comments and their declaration context, plus all 15 public utility exports. The nine-module import graph is acyclic. Public tokenizer/segmenter function identities and #173 built-in routing remain exact; there are no wrappers, duplicated mutable caches or parser policy changes. All other source and tests match the landed base. All 3,032 tests and seven local checks pass with 98.11% branch coverage; all 39 files stayed unchanged through validation and normal commits. The final merge of main changed ancestry only, retaining the exact validated tree; no duplicate runtime run is claimed. Please inspect ownership/import edges, exports and identity, type/build behavior and actual source movement. The complete prior parser review and documented heuristic limits remain applicable to unchanged declarations. Current-head declared-toolchain CI runs separately; the preserved local package installation denial was not retried or bypassed. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Separate text preprocessing from numeric utilities while preserving the public API and the fully integrated parser behavior.
Changes
src/sentence-segment.ts, Treebank tokenization intosrc/tokenize.ts, and shared character and quotation helpers intosrc/text.ts.src/utils.tsand re-export the existing text API from that module. Existing imports and the function identities used to select built-in preprocessing remain valid.AGENTS.md.This is a declaration-preserving extraction from the final #170 implementation. All 305 declarations, 187 comments and their declaration context, and the public export set are retained. The nine-module import graph is acyclic. The scoring and preparation modules, all tests, mutable default options, dependencies and workflows remain unchanged. Runtime dependencies remain zero.
The shared text module owns the modifier-footnote helpers used by quotation-opening context so both tokenizer and sentence parser use the same definitions. Scoped-question and sentence-boundary state stay in the sentence module. No wrapper functions, duplicate caches, or parser policy changes are introduced.
Validation
Exact head
8723c42419ff9ed5e8f3a00e62e3bd3786d95f33, tree00a180013eafa04354ef86472ce712ce9c643b44: 3,032 tests pass, with 98.11% branch coverage. Source/test types, strict installed Biome, Markdown formatting, build, and packed CJS/ESM/TypeScript consumers pass. All 39 tracked-file hashes remain unchanged through validation and normal commit hooks.The final merge of main changes ancestry only and preserves every file from the completed 3,032-test validation. No duplicate runtime run is claimed.
Independent retention and source review verified declaration bodies and comment placement, helper dependencies, direct public function identity, and default preprocessing routing. The original constrained-heap and repeated-input tests remain in the unchanged suite. Existing heuristic limits are preserved.
Current-head Codex code/security and Copilot reviews, including suppressed findings, and exact declared-toolchain CI are requested separately. Local Biome 2.5.10 and Node types 24.13.3 remain older than declared 2.5.13 and 24.13.4 after the preserved installation denial; no alternate install or configuration workaround was used.