feat: complete VLDB tasks and optimize FTS hot paths - #1142
Open
xingyanchen-0609 wants to merge 17 commits into
Open
feat: complete VLDB tasks and optimize FTS hot paths#1142xingyanchen-0609 wants to merge 17 commits into
xingyanchen-0609 wants to merge 17 commits into
Conversation
Implement the two Document AI functions for the ai_funcs test suite: LOAD_FILE(location_name, file_name) -> BLOB - new scalar sys expr ObExprLoadFile (T_FUN_SYS_LOAD_FILE): looks up the LOCATION schema by name, joins its URL with the file name and reads the file through ObBackupIoAdapter, returning the content as a binary lob AI_SPLIT_DOCUMENT(content [, parameters]) table function - new non-reserved keyword + grammar rule producing T_AI_SPLIT_DOCUMENT_EXPRESSION, resolved as a JSON_TABLE-family table item (OB_AI_SPLIT_DOC_TABLE_TYPE) with four fixed output columns chunk_id / chunk_offset / chunk_length / chunk_text - AiSplitDocTableFunc splits the document per parameters json: type text|markdown (default markdown), by word|sentence (default word), max units per chunk (default 256), overlap for sliding windows; markdown mode sections the text at '#' heading lines and prefixes each chunk with its section heading - spec serialization, printer (view expansion) and rescan support Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement the Custom Dictionary feature for the IK fulltext parser: - CREATE TABLE ... FULLTEXT_DICT='Y' table option: new FULLTEXT_DICT non-reserved keyword and table option, accepted for CREATE TABLE (value must be 'Y'/'N', rejected in ALTER TABLE like ORGANIZATION). - ALTER SYSTEM REFRESH FULLTEXT DICT [db.]table statement: new DICT keyword, T_REFRESH_FULLTEXT_DICT parse node, resolver, stmt and executor wired through the standard system-command chain. Accepts the unquoted db.table / table forms and the double-quoted string forms. Custom dictionaries are re-read from the dictionary table on every parser instantiation, so the refresh command has nothing to invalidate and newly written words become visible immediately. - Runtime: ObFTParserProperty now really extracts dict_table / stopword_table / quantifier_table values from the parser-properties JSON (deep-copied into owned buffers), ObFTParseHelper passes them into ObFTIKParam, and ObIKFTParser::init_dict loads a dictionary table (SELECT LOWER(word) ... ORDER BY LOWER(word) COLLATE utf8mb4_bin via inner SQL, then trie -> DAT -> ObFTCacheDict) when the configured table differs from the built-in one, REPLACING the corresponding built-in dictionary. An empty dictionary table maps to the new ObFTEmptyDict which never matches. Covers tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test: verified byte-exact against the recorded .result (custom words match, words missing from the custom dict degrade to single characters and no longer match, dynamic INSERT + REFRESH takes effect for new rows), and the built-in IK path (no dict_table property) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Optimize the IK/beng full-text tokenization paths that dominate fulltext index build, TOKENIZE() and MATCH query-token segmentation: - Share the three built-in IK dictionaries (main/quantifier/stopword) process-wide in ObFTParsePluginData: they are immutable, so building the range dict once and borrowing it from every parser instance removes the per-document KV-cache lookups and dictionary object rebuilds that used to run on every tokenization. Custom dictionary tables (FULLTEXT_DICT='Y') keep the per-instance build so REFRESH semantics are unchanged. - Classify each character with a single unicode decode in ObFTCharUtil::do_classify instead of re-decoding for every category check (up to 4 decodes per CJK char before). - Fetch char, length and type in one TokenizeContext::current_char_and_type call in the two per-character hot loops. - Store IK segmenters in a fixed array instead of an allocator-backed list. - Skip the per-token charset conversion (and its allocation) in ObStopWordChecker::check_stopword when source and stopword charsets match. All ai_funcs mysqltest cases still pass byte-exact and tokenization output is unchanged; local fts_large_bench shows build_ik_all -19%, tokenize_ik -37%, tokenize_beng -22% at identical hit counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable the (previously commented out) check_need_calc_match_score pass in the optimizer: when a MATCH expr is only used as a boolean predicate (e.g. SELECT COUNT(*) ... WHERE MATCH(...) AGAINST(...)), skip the BM25 relevance infrastructure entirely - no total-doc-count aggregation scan, no per-token doc-count aggregation, no forward-index scans and no BM25 evaluation. The match filter then only checks a constant positive hit score. Make the relevance-less plan shape actually executable in the sparse retrieval runtime (these latent null derefs are why the pass had been disabled): - only create/init the doc-cnt agg iter and BM25 estimation context when a doc agg ctdef was generated, and keep the estimator uninitialized when the estimation context is invalid - allocate the skip bit vector unconditionally in the token iter and skip token-doc-cnt estimation without aggregation iters - template TAAT scan ranges from the inverted index scan param when there is no aggregation param, and count dimension hits as 1.0 without a relevance expr - functional lookup / match score output / fts index merge branches always calculate relevance since the score itself is their output All ai_funcs mysqltest cases pass byte-exact, MATCH result sets (natural language, boolean mode, score projection, order-by-score, functional lookup) are unchanged, and fts_large_bench hit counts are identical with query latency down 15-20% locally on top of the tokenization gains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…60367468) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- TokenizeContext::prepare_next_char decodes length, code point and char type in a single utf8 pass (well_formed_len + re-decode before), and caches the code point so letter/quantifier processors check connectors and cn-numbers without decoding the same bytes again - ObIKArbitrator::output_result uses the same single-pass decode for both its scan loops and reuses the decoded code point for the single-cjk ignore check; utf8-encoded surrogates keep their legacy classification - ObAddWord::groupby_word inserts-or-accumulates with one hash lookup via set_or_update instead of get + set - ObFTIndexRowCache keeps one reusable ObFTWordMap across documents so index build no longer creates/destroys a hash map per doc Verified: ai_funcs (load_file/ai_split_document/ik_custom_dict) all pass, fts_large_bench hit counts identical (cn 8001 / beng 11000 / mixed 7332 / limit 20), tokenize_ik 1.00->0.36ms on the local rig.
TOKENIZE() without explicit properties reforms the same default property JSON for a given parser on every statement (json init/parse/rebuild/ serialize). Cache the reformed JSON per parser inner name in a small process-wide table; statements with explicit properties keep the original path. Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
- ObFTParseHelper::init parsed the same property JSON on every call (per statement for TOKENIZE, per helper for index build); cache the parsed ObFTParserProperty keyed by (parser name, property json) in a small process-wide table, falling back to the original parse path when the cache is full or the key is oversized - TOKENIZE per-statement word map sizes buckets at len/4 instead of len/2; load factor stays below one for typical token densities Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
The TAAT doc-dedup hash maps were created with 10 buckets while each partition is designed to hold about 1000 docs (and a single partition holds every hit when relevance calculation is skipped), which forced hundred-deep collision chains on every per-hit upsert. Size buckets to 2x the designed partition size, upsert with one set_or_update pass instead of get + set, and use NoPthreadDefendMode since the maps are only touched by the executing query thread. Verified: ai_funcs 3/3 pass, bench hit counts unchanged (cn 8001 / beng 11000 / mixed 7332 / limit 20).
|
陈星岩 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 14.68 / 100 mean_improvement: 7.34% full_score_improvement: 50.00% build_improvement: 5.51% build_ik_all_sec: baseline=35.2836, current=32.662, improvement=7.43% build_ik_content_sec: baseline=28.3764, current=26.906, improvement=5.18% build_beng_en_sec: baseline=14.7578, current=14.181, improvement=3.91% tokenize_improvement: 2.86% tokenize_ik_avg_ms: baseline=0.76478, current=0.711, improvement=7.03% tokenize_beng_avg_ms: baseline=0.42262, current=0.4282, improvement=-1.32% query_improvement: 13.65% query_cn_avg_ms: baseline=16.6628, current=13.633, improvement=18.18% query_beng_avg_ms: baseline=24.3042, current=20.205, improvement=16.87% query_mixed_avg_ms: baseline=17.5593, current=14.2629, improvement=18.77% query_limit_avg_ms: baseline=16.2334, current=16.1056, improvement=0.79% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-18 08:21:33 label: vldb-ci-29636404879-1 git_head: d58411d git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2738 select1_stdev_ms: 0.0154 raw_load_sec: 1.493 raw_load_rows_per_sec: 13395.8 build_ik_all_sec: 32.662 build_ik_content_sec: 26.906 build_beng_en_sec: 14.181 build_total_sec: 73.762 ---------------------------------------- tokenize_ik_avg_ms: 0.7110 tokenize_ik_median_ms: 0.7114 tokenize_ik_stdev_ms: 0.0017 tokenize_beng_avg_ms: 0.4282 tokenize_beng_median_ms:0.4276 tokenize_beng_stdev_ms: 0.0017 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 13.6330 query_cn_stdev_ms: 0.0017 query_beng_hits: 11000 query_beng_avg_ms: 20.2050 query_beng_stdev_ms: 0.0604 query_mixed_hits: 7332 query_mixed_avg_ms: 14.2629 query_mixed_stdev_ms: 0.0094 query_limit_hits: 20 query_limit_avg_ms: 16.1056 query_limit_stdev_ms: 0.0130 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is an independent team submission based on
huangda99/DB2026_seekdb:vldb_2026.LOAD_FILEandAI_SPLIT_DOCUMENTfor the Document AI task.ALTER SYSTEM REFRESH FULLTEXT DICT.Validation
100/100, IK Custom Dictionary100/100, FTS benchmark11.98/100.ik_custom_dictmysqltest:ok.observer/seekdbbuild completed after the BEng changes.5.502s -> 5.100s(about7.31%faster).0.1223ms -> 0.1192ms(about2.53%faster); focused A/B reached0.1153ms.8001, BEng11000, mixed7332, LIMIT20.The benchmark is timing-sensitive, so the official VLDB CI result is authoritative.
Team: 大海好多水