Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #543.
Cause
In
csv-parse, the delimiter and record-delimiter matchers (__isDelimiter,__isRecordDelimiter,__autoDiscoverRecordDelimiter) compare their byte pattern against the raw buffer at the current position, with no constraint on where the match starts. Forutf16le, a code unit is 2 bytes, so a delimiter that "matches" at an odd absolute offset actually starts halfway through a code unit:Valid characters are then broken into empty records or fields, losing the original value.
Fix
Delimiters may only match on a code unit boundary. A small
__isByteAligned(pos)helper bails the three matchers out whenstate.bufBytesStart + posis not even and the encoding isutf16le—bufBytesStartalready tracks the absolute stream offset (BOM length included) across chunk reconciliations, so alignment holds for chunked and streamed input too.Field delimiters, record delimiters, and auto-discovery are covered. The other byte matchers (
quote,escape,comment, trimable characters) have the same theoretical exposure and are left for a separate change.Tests
Three cases from the issue in
test/option.encoding.ts(\u0a01\u0100withrecord_delimiter: "\n",\u0d01\u0100withrecord_delimiter: "\r",\u2c01\u0100withdelimiter: ","): each parses to[[value]]instead of being split. All three fail onmasterand pass with this change; the full csv-parse suite is green (563 passing,test/samples.jsexcluded — it references asamplespath that does not resolve on Windows, unchanged onmaster), andtsc --noEmitis clean.