Repository navigation
Keep per-format routing and stream traits in the chat-format registry - #1460
Merged
Merged
Conversation
Characterization golden captured from main's per-format tables before they move into the format registry (#362).
The stream parser's tool-call opening switch and thought/envelope tables, and the engine's schema, lazy-grammar, generic, tool_choice=none and canonical-JSON-Schema tables move into a ChatFormatTraits value on each ChatFormatRegistration. The pinned golden is unchanged. Refs #362.
Fails if a per-format table reappears in the stream parser or engine, and pins the Firefunction v2 leading space and the Kimi K3 end-of-message marker that surviving mutants dropped. Refs #362.
Contributor
|
Chat app preview removed for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ChatFormatRegistrationas aChatFormatTraitsvalue (lib/src/core/template/chat_format_registry.dart).traitsis a required argument, so a format cannot be registered without them.chat_completion_stream_parser.dart: the 25-arm_toolCallOpeningForswitch,_keepsThoughtTagsAsContent,_parsesCallsInsideThoughtsand_mayEmbedToolEnvelopeAfterContent.chat_template_engine.dart:_schemaDisabledFormats,_requiredKeepsLazyFormats,_toolCallGenericFormatsand the Gemma branch after it (now oneroutesThroughGenerictrait),_toolChoiceNoneContentOnlyFormats, and the canonical JSON Schemasupportedset together with the incomplete-tool-call switch in_parse. Those two always listed the same six formats; they are now oneleavesIncompleteToolCallcheck, andqualifiesJsonSchemaToolsis true exactly when that check is set.test/fixtures/chat_format_traits.jsonwas committed first (d0592cc86). At that commit its test reads main's private tables throughdart:mirrorsand passes against unchangedlib/. The refactor commit switches the reader totraits, and the golden is byte-identical at head. The golden also pins public-API probes that don't depend on how the tables are stored: parse outcomes for 10 incomplete-tool-call outputs × 45 formats (with tool parsing on and off), and render routing for every detection trigger under 8 tool/schema/tool_choice combinations.ChatFormatset or collection literal, a format switch arm, or a statement naming three or more formats comes back in the stream parser or engine. Two near-miss detector tests close the mutants that survived the #1457 audit (the leading space in' functools['and Kimi K3 without<|end_of_msg|>).Refs #362 (slice 2 of the 0.17.0 Backend SPI cleanup, #922).
Production-readiness scope
Completeness checklist
PR type guidance
Test Plan
dart run tool/prepare_workspace.dart(clean tree afterwards)dart format --output=none --set-exit-if-changed .dart analyzedart run tool/testing/check_platform_boundaries.dartdart test -p vm -j 1 --exclude-tags local-only(macOS arm64, isolatedTMPDIR): 8,547 passed, 5 skipped (an earlier run alongside the parity llama.cpp build failed onenative_symbol_integration_test.dartcase, which passed alone 26/26 and in this clean rerun)dart test -p chrome --exclude-tags local-only test/unit/core/template test/unit/core/engine/chat_completion_stream_parser_test.dart: 1,049 passed./tool/testing/run_template_parity_suites.sh(llama.cppb10549corpus): detection 74, diagnostics 88, unit 1,254, upstream parser suites 10, compiled grammar 52, all pass (the grammar step hit a Dart SDK native-assetsotoolcrash while a concurrent full VM run shared.dart_tool, and passed when rerun alone)Mutation checks: each mutant was applied alone at head and then reverted. 21/22 were killed. Script and log:
llamadart-evidence/s2-chat-format-traits/{mutate.py,mutants.log}.hasTools; incomplete-call check ignoresparseToolCalls; generic routing ignores tools' functools['without space; Kimi K3 without<|end_of_msg|>Set<ChatFormat>added back to the engineMatrix Evidence
b10549templatesrun_template_parity_suites.sh; render, parse and stream code paths read the same values as before184453043/0a33025e9llamadart-evidence/pr-1460/evidence-1460-1844530.json;high_risk_readiness.dartexits 2.Review Notes
184453043(independent auditor session). P3s: guard scan misses ChatFormat-keyed map literals, two-format helpers andcasearms; pre-existing gap on genericthinkingForcedOpenin the incomplete-call check.184453043: 32 pass, 5 skipped, 0 failing (run 40509270176).High-risk regression review
structuredOutput:lib/src/core/template/;backendRuntime:lib/src/core/engine/)184453043/0a33025e9ChatTemplateEngine.render/parse; trait values are read from the registrations the engine and stream parser use.b10549corpus and the golden.