Skip to content

Make string-length, substring and translate count characters - #139

Merged
zhengchun merged 1 commit into
antchfx:masterfrom
youdie006:rune-correct-string-functions
Sep 15, 2026
Merged

zhengchun merged 1 commit into
antchfx:masterfrom
youdie006:rune-correct-string-functions

Conversation

@youdie006

Copy link
Copy Markdown
Contributor

Follow-up to #138, where I noted this and offered to open it separately.

XPath 1.0 4.2 defines these three in characters; all three index by byte. stringLengthFunc's own doc comment at func.go:575-576 already says "number of characters".

expression this repo libxml2 2.14.6
string-length("héllo") 6 5
string-length("日本語") 9 3
substring("héllo",1,3) "hé" "hél"
substring("日本語abc",2,2) "\x97\xa5" (not valid UTF-8) "本語"
translate("abc","ab","áé") "ác" "áéc"
translate("日本語","日語","ab") mojibake "a本b"

normalizespaceFunc (func.go:469) already converts to []rune for exactly this reason — you added that in be94ad6 to close #32. These three are the remaining byte-indexed string functions. substring-before/substring-after (func.go:564-571) are correct as they are, because strings.Index returns a boundary-aligned offset, so they are not touched.

The three have to move together. Fixing string-length alone breaks the common idiom, because the length is then in characters while the slice is still in bytes:

string-length("héllo") substring("héllo",1,string-length("héllo"))
master 6 "héllo"
string-length fixed alone 5 "héll"
this PR 5 "héllo"
libxml2 5 "héllo"
Verification

Base 865cb21. Assertions were added as rows to the three existing table tests, so the filter was checked by counting === RUN lines — 3 on every row, never 0.

row func.go md5 RUN exit failing test
GREEN 8c32ba35… 3 0 —
RED (master) fe065ef3… 3 1 all three
revert string-length only 675747ca… 3 1 Test_func_string_length
revert substring only 05e43b23… 3 1 Test_func_substring
revert translate only 840aa414… 3 1 Test_func_translate
over-correct RuneCountInString(v)+1 fa195add… 3 1 Test_func_string_length
over-correct clip to len(rs)+2 5f537834… 3 1 Test_func_substring (on a pre-existing row)
over-correct start >= 29c88b9c… 3 1 Test_func_substring
over-correct translate i+1 74048c27… 3 1 Test_func_translate

The three reverts are disjoint, so each part of the change is independently pinned, and the over-corrections fail, so the rows do not stop one short.

The reference column is libxml2 2.14.6 via lxml, checked on 12 classes of input before any assertion was written, so the new rows encode its behaviour rather than my reading of the spec.

Gates, as .github/workflows/*.yml write them: go test ./... ok; go test -covermode atomic -coverprofile=coverage.out ./... ok, coverage 86.9%; go vet ./... exit 0; gofmt -l clean on both changed files. (gofmt -l . also lists func_go110.go and func_pre_go110.go; both are already unformatted at 865cb21, and neither workflow runs gofmt.) Run with go1.26.3 rather than the 1.16 pinned in coverage.yml; unicode/utf8 and []rune both long predate go.mod's go 1.14.

Two costs worth naming rather than hiding: substring and translate now allocate a []rune, which I have not benchmarked — be94ad6 was explicitly a performance commit, so say the word if you want numbers. And this changes answers for non-ASCII input, so it is a behaviour fix, not a pure internal one.


Disclosure: found and written with the help of Claude (an AI assistant). Every number above is from runs on my machine.

XPath 1.0 4.2 defines these in characters; all three index by byte, so
string-length("日本語") is 9 and substring("日本語abc", 2, 2) returns a
byte slice that is not valid UTF-8. stringLengthFunc's own doc comment
already says "number of characters".

The three move together: fixing string-length alone turns the common
idiom substring(s, 1, string-length(s)) from "hello" into a truncated
string, because the length is then in characters but the slice is in
bytes.

normalizespaceFunc already converts to []rune for the same reason.
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 84.623% (+0.02%) from 84.602% — youdie006:rune-correct-string-functions into antchfx:master

@zhengchun
zhengchun merged commit 551dd37 into antchfx:master Sep 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

normalize-space does not conform to the W3C recommendation

3 participants