Skip to content

fix(csv-parse): only match delimiters on utf16le code unit boundaries - #545

Open
nimbly1 wants to merge 1 commit into
adaltas:masterfrom
nimbly1:fix/utf16le-unaligned-delimiters
Open

nimbly1 wants to merge 1 commit into
adaltas:masterfrom
nimbly1:fix/utf16le-unaligned-delimiters

Conversation

@nimbly1

@nimbly1 nimbly1 commented Oct 26, 2026

Copy link
Copy Markdown

Fixes #543.

Cause

In csv-parse, the delimiter and record-delimiter matchers (__isDelimiter, __isRecordDelimiter, __autoDiscoverRecordDelimiter) compare their byte pattern against the raw buffer at the current position, with no constraint on where the match starts. For utf16le, a code unit is 2 bytes, so a delimiter that "matches" at an odd absolute offset actually starts halfway through a code unit:

// "\u0a01\u0100" in utf16le is 01 0a 00 01
// the LF pattern 0a 00 is "found" at offset 1, splitting the value

Valid characters are then broken into empty records or fields, losing the original value.

Fix

Delimiters may only match on a code unit boundary. A small __isByteAligned(pos) helper bails the three matchers out when state.bufBytesStart + pos is not even and the encoding is utf16le — bufBytesStart already tracks the absolute stream offset (BOM length included) across chunk reconciliations, so alignment holds for chunked and streamed input too.

Field delimiters, record delimiters, and auto-discovery are covered. The other byte matchers (quote, escape, comment, trimable characters) have the same theoretical exposure and are left for a separate change.

Tests

Three cases from the issue in test/option.encoding.ts (\u0a01\u0100 with record_delimiter: "\n", \u0d01\u0100 with record_delimiter: "\r", \u2c01\u0100 with delimiter: ","): each parses to [[value]] instead of being split. All three fail on master and pass with this change; the full csv-parse suite is green (563 passing, test/samples.js excluded — it references a samples path that does not resolve on Windows, unchanged on master), and tsc --noEmit is clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UTF-16LE delimiters match at unaligned byte offsets

1 participant