[TheFabbest]

Summary

Relates to #464

Adds InternationalTokenizer, selectable with --tokenizer international. It splits text into words with ICU's icu::BreakIterator, which makes it possible to tokenize languages written without spaces (Chinese, Japanese, Thai and more) and mixed-language text.

ICU is an optional dependency. Without it, PISA builds and works normally and cmake prints a warning, but --tokenizer international fails at runtime with an error.
Most Linux distributions already have the ICU runtime installed, but PISA needs the development files (libicu-dev on Debian/Ubuntu, libicu-devel on Fedora), which may be missing, for example in Docker images. As far as I know, macOS should work with -DICU_ROOT=$(brew --prefix icu4c), but I couldn't test it.

Behavior

For InternationalTokenizer:

• Words are found with ICU's word break rules (UAX 29 + dictionary-based on top). This dictionary-based algorithm explicitly supports languages such as Chinese, Japanese, and Thai, as well as mixed scripts.
• Whitespace, symbols and punctuation segments are dropped.
• Text is case-folded (full Unicode case folding, e.g. Straße -> strasse). This is standard, but can be changed easily to lowercase only. Accents are kept.
• Apostrophes inside words are kept (shouldn't -> shouldn't). Dotted words stay whole, only a trailing . is dropped (U.S.A. -> u.s.a, himself.image -> himself.image).
• Locale-independent, this is relevant for some languages (e.g. Turkish) that have locale-specific case folding rules. I opted for the locale-independent behavior for deterministic reproducibility.
• Different versions of ICU may have different dictionaries, so the exact tokenization of CJK text may vary; this hurts reproducibility, but it is the best we can do without shipping our own dictionary.

Dependency handling

ICU is large, but is available as a standard system package on common platforms. PISA does not vendor it, so unlike the PISA_SYSTEM_* options (which choose between a system and a vendored copy of a dependency), PISA_ENABLE_ICU simply enables or disables an optional feature. The default of ON follows the same convention as the vendored boost-cmake's BOOST_LOCALE_ENABLE_ICU_BACKEND ("enable if available", default ON). The table below summarizes the behavior:

ICU available at configure time	PISA_ENABLE_ICU	Result
yes	ON (default)	PISA_HAS_ICU=1 is defined, ICU is linked, tokenizer works
no	ON (default)	cmake warns "ICU not found"; builds normally; tokenizer throws std::runtime_error
yes or no	OFF	builds normally; tokenizer throws std::runtime_error

ICU is already probed optionally by the existing build via external/boost-cmake/libs/locale.cmake (find_package(ICU ...) without REQUIRED, falling back to iconv). So this PR does not add a new required dependency, but it is the first time PISA's own code uses ICU.

No ICU headers are exposed by PISA’s public headers: InternationalTokenStream is declared in tokenizer.hpp like the other streams, but its ICU state lives behind a pimpl defined in tokenizer.cpp. Since PISA is a static library, binaries linking it when built with ICU also link ICU.

ICU is widely used and well-maintained, and is the de facto standard for Unicode text processing.

Testing

Environment: PopOS! 24.04 (ubuntu-based), GCC 13.3, ICU 74.2 locally (was already installed).
Also an Ubuntu 24.04 docker container with no ICU headers installed.

• All four configurations (ICU installed or not, PISA_ENABLE_ICU ON or OFF) were checked.
• Build and tests pass in all four configurations.
• In the container without ICU headers, -DPISA_COMPILE_HEADERS=ON passes, confirming that tokenizer.hpp doesn’t leak ICU includes.
• Tested built parse_collection tool with all three tokenizers and manually verified that the output is as expected.
• An informal measurement (samples from the Coca corpus) shows that the new tokenizer is about 2x slower than the existing EnglishTokenizer on English text, with little difference in memory usage.

Notes

1. Default for PISA_ENABLE_ICU. I chose ON (auto-detect) to match the existing Boost.Locale probe. If you prefer optional features to be strictly opt-in, flipping it to OFF is a one-line change.
2. Docs. Documentation probably needs to be updated to mention the new tokenizer and its ICU dependency.
3. CI. If you want to test the new tokenizer in CI, the CI image needs the ICU development package. Otherwise, a simpler test will be run that just checks that the tokenizer can be constructed and throws an error when ICU is not available.
4. Thread safety. The thread safety of the new tokenizer has not been thoroughly tested. However, each stream owns its own icu::BreakIterator and icu::UnicodeString, with no shared mutable state. This suggests it is safe.
5. Windows and macOS. I can try to build and test on Windows 11 if you want, but I don't have a Mac.
6. Move-only stream. Unlike the other streams, InternationalTokenStream is not copyable. Streams are only handed out as std::unique_ptr<TokenStream>, so copying isn’t needed, and a correct copy would have to re-bind the ICU break iterator to the copied text. I can add copy support if you prefer consistency.