tokenizer-mismatch-module-page

[AndreasAbdi]

{
"project": "Tokenizer Mismatch Canonical Page",
"branchName": "tokenizer-mismatch-module-page",
"description": "Add one canonical English docs page for tokenizer mismatch so readers can understand why the wrong tokenizer, special-token handling, or prompt-format assumptions cause practical failures or degraded model behavior, and find that explanation through the tokenizer-family discovery surfaces.",
"context": {
"customerAsk": "Customer ask alignment: add a canonical English page for tokenizer mismatch, which is explicitly called for in the roadmap bundle but is still absent from the shipped docs tree. Treat this as one narrow tokenization-module slice on current main. Scope: create the correct canonical page kind and route for tokenizer mismatch according to the current standards, including any missing registry record, page.mdx, messages/en.json, and assets.json; connect it to existing tokenization, model, and serving pages such as tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, and embedding; and add only the minimum focused validation/tests needed for the touched surfaces. The prose should explain in simple language what tokenizer mismatch is, why it causes practical failures or degraded behavior, and where readers will see it in real model usage. Keep the slice English-only and avoid broad tokenization-family rewrites. Acceptance criteria: the canonical tokenizer-mismatch page exists on current main, is registry-backed and discoverable, and focused quality gates for the touched content surfaces pass.",
"problem": "The roadmap already calls for a tokenizer mismatch page, but the shipped docs tree still does not provide one. Readers can learn what tokens, embeddings, special tokens, BPE, WordPiece, and SentencePiece are, yet they do not have a dedicated page explaining the practical failure mode that appears when a model is paired with the wrong tokenizer, different special-token handling, or text that is segmented differently from what the model was trained to expect. That leaves a discovery gap across the tokenizer family and nearby model-input or serving paths.",
"solution": "Publish tokenizer-mismatch as a canonical module page in the tokenization family, backed by a valid module.tokenizer-mismatch registry record if missing, English-localized messages, and focused discovery metadata that ties it to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, embedding, and representative model or serving pages where those relationships are already justified. Add only narrow behavioral validation for the route, registry linkage, and discovery surfaces touched by this page."
},
"acceptanceCriteria": [
"A published /docs/modules/tokenizer-mismatch page exists on current main with canonical module-page structure and plain-language copy explaining what tokenizer mismatch is, why it matters, and where readers see it in practice.",
"A valid published registry record exists for module.tokenizer-mismatch with tokenization-family classification, aliases, tags, summary metadata, and related-document fields aligned with project docs and the data model.",
"The page and registry-backed discovery metadata connect tokenizer-mismatch to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, and embedding, plus only those representative model or serving pages whose relationship is already justified by shipped content.",
"Search documents and at least one relevant tag or related-doc discovery surface can return or surface the canonical tokenizer mismatch page for representative queries such as tokenizer mismatch, wrong tokenizer, or special token mismatch.",
"The landed slice remains English-only and does not reopen unrelated tokenization-family rewrites, locale infrastructure, or non-touched content bundles.",
"Quality gate: make typecheck, make lint, and focused tests for the touched page, messages, registry, and discovery behavior pass."
],
"userStories": [
{
"id": "tokenizer-mismatch-module-page-001",
"title": "Establish tokenizer mismatch as a first-class tokenization module",
"description": "As a reader browsing tokenizer topics, I want tokenizer mismatch classified as a canonical tokenization page so that related docs and taxonomy surfaces place it next to the right neighboring topics.",
"acceptanceCriteria": [
"A published registry record exists for module.tokenizer-mismatch with stable id, canonical slug tokenizer-mismatch, moduleType: tokenizer, tokenization-family grouping, valid aliases, tags, summary fields, and reviewer-checkable related-document metadata.",
"Registry classification follows docs/documentation-site-pages-needed.md by treating tokenizer mismatch as a module-family tokenization page rather than a broad concept page or glossary entry.",
"Registry relationships connect tokenizer-mismatch to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, and embedding, plus representative model or serving pages only where those relationships are concrete and non-duplicative.",
"The slice does not broaden into unrelated tokenization taxonomy cleanup outside the metadata required to land this page cleanly.",
"Typecheck passes",
"Tests pass"
],
"priority": 1,
"passes": true,
"notes": ""
},
{
"id": "tokenizer-mismatch-module-page-002",
"title": "Publish the canonical tokenizer mismatch explainer page",
"description": "As a technical layperson learning how models consume text, I want a dedicated tokenizer mismatch page so that I can understand why using the wrong tokenizer causes practical failures or degraded behavior.",
"acceptanceCriteria": [
"A canonical module page exists at /docs/modules/tokenizer-mismatch with matching frontmatter, messages/en.json, and any required local assets.json.",
"The page opens with one folded openingSummary and explains tokenizer mismatch in plain language before narrowing into model-specific or serving-specific examples.",
"The page explains that a mismatch can come from using the wrong vocabulary, different token-splitting rules, incompatible special tokens, or prompt formatting assumptions that do not match the model's training or serving setup.",
"The page explains reader-visible consequences such as shifted token counts, broken chat-template boundaries, degraded completions, or failures around special tokens and embeddings, without turning into a benchmark page or implementation tutorial.",
"Customer-facing copy is English-only, understandable in isolation, and free of process or authoring meta language.",
"Typecheck passes",
"Tests pass",
"Verify in browser using the Browser plugin"
],
"priority": 2,
"passes": true,
"notes": ""
},
{
"id": "tokenizer-mismatch-module-page-003",
"title": "Make tokenizer mismatch discoverable from nearby tokenization and model-input paths",
"description": "As a reader who searches for tokenization problems or lands on nearby pages, I want discovery surfaces to route me into tokenizer mismatch so that I can find the explanation without already knowing the slug.",
"acceptanceCriteria": [
"Search documents include /docs/modules/tokenizer-mismatch with title, aliases, tags, and core body terms that let representative queries such as tokenizer mismatch, wrong tokenizer, prompt token mismatch, and special token mismatch find the page.",
"The canonical page renders tag and related-doc surfaces that connect it to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, and embedding, plus representative model or serving pages where those canonical targets exist and are contextually justified.",
"At least one neighboring shipped discovery surface or related-doc path can lead readers into tokenizer-mismatch without typing the route directly.",
"Browser-visible rendering shows the page title, summary, tags, and related-doc links without missing-content placeholders or broken links.",
"Typecheck passes",
"Tests pass",
"Verify in browser using the Browser plugin"
],
"priority": 3,
"passes": true,
"notes": ""
},
{
"id": "tokenizer-mismatch-module-page-004",
"title": "Add focused validation for the tokenizer mismatch page contract",
"description": "As a maintainer, I want targeted automated proof for the tokenizer mismatch slice so that route, registry, message, and discovery regressions are caught without unrelated test expansion.",
"acceptanceCriteria": [
"Focused validation or test coverage confirms the tokenizer-mismatch page route, registry record, and default English messages resolve together.",
"Coverage asserts at least one page-specific related-doc, tag, or search expectation for tokenizer-mismatch.",
"Coverage fails if the page loses its canonical route, registry linkage, tokenization-family classification, or representative discovery visibility.",
"Validation stays scoped to observable behavior for this page slice and does not require unrelated inventory snapshots, locale-manifest churn, or repository-wide taxonomy counts.",
"Typecheck passes",
"Tests pass"
],
"priority": 4,
"passes": true,
"notes": ""
}
]
}

[AndreasAbdi]

Completed story tokenizer-mismatch-module-page-004. Added a focused tokenizer-mismatch runtime contract test that verifies the canonical route, published registry classification, English messages, search aliases, and tokenization tag discovery together. Also fixed mergeability-only test contracts revealed by the full suite by updating the module sidebar inventory plus bundled graph/table runtime inventories for the new tokenizer-mismatch assets. Local verification for this iteration: make lint, make typecheck, make validate-data, make test.

[AndreasAbdi]

BLOCKING

Primary findings:

1. src/content/registry/modules/tokenizer-mismatch.json gives this page releaseDate: "2019-02-14", GPT-2-report authors, and sourceId: "citation.gpt-2-report". That flows into ModuleAtAGlance and the rendered page now states that tokenizer mismatch was "Released" in February 2019 by Alec Radford et al. with the GPT-2 report as the source. That is not a correct attribution for this topic; tokenizer mismatch is being documented here as a failure mode, not as a discrete module introduced by that paper. This is a correctness issue and also fails the docs-writing citation rule because the cited source does not support the page's chat-template / serving-stack claims.
2. gh pr view --json mergeable,mergeStateStatus currently reports mergeable: CONFLICTING and mergeStateStatus: DIRTY, so this PR is not mergeable as-is even if the content issue is fixed.

Quality checks:

• make test: PASS
• make lint: PASS
• make typecheck: PASS
• Browser verification against a local production build on http://127.0.0.1:3456/docs/modules/tokenizer-mismatch: PASS for route rendering, related-doc links, and tag discovery. Only console issue was favicon.ico 404.
• Live GitHub checks: gh pr checks 123 reports no checks on the current head.

Project acceptance criteria:

1. PASS — /docs/modules/tokenizer-mismatch exists and the page explains the topic in plain language.
2. FAIL — the registry record exists, but the summary metadata is not aligned with the docs-writing standard because it falsely attributes release date/authorship/source to GPT-2.
3. PASS — the page and discovery metadata connect to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, embedding, and gpt-3.
4. PASS — search and tag discovery surface the page for representative tokenizer-mismatch queries.
5. PASS — the slice stays English-only and does not broaden into unrelated taxonomy work.
6. PASS — the local quality gates I ran (make test, make lint, make typecheck) passed.

User story acceptance criteria:
tokenizer-mismatch-module-page-001

1. FAIL — the registry record is present, but the release/authorship/source metadata is not reviewer-correct.
2. PASS — classification is module with moduleType: tokenizer and moduleFamily: tokenization.
3. PASS — related registry links match the intended tokenizer neighbors plus gpt-3.
4. PASS — the diff stays scoped.
5. PASS — typecheck passed locally.
6. PASS — tests passed locally.
   Behavioral assertion check: PASS — this story includes observable registry/discovery behavior.

tokenizer-mismatch-module-page-002

1. PASS — canonical route, frontmatter, messages, and assets are present.
2. PASS — the page opens with one folded openingSummary and browser/runtime tests confirm the shell structure.
3. PASS — the page explains wrong vocabulary, split rules, special tokens, and prompt wrappers.
4. PASS — the page explains reader-visible consequences.
5. PASS — copy is English-only and avoids process/meta prose.
6. PASS — typecheck passed locally.
7. PASS — tests passed locally.
8. PASS — browser verification completed.
   Behavioral assertion check: PASS.

tokenizer-mismatch-module-page-003

1. PASS — search documents and live search return the page for representative mismatch queries.
2. PASS — related-doc and tag surfaces render the expected neighbors.
3. PASS — the tokenization tag landing page leads to the page without typing the direct route.
4. PASS — browser rendering shows title, summary, tags, and related links without missing placeholders.
5. PASS — typecheck passed locally.
6. PASS — tests passed locally.
7. PASS — browser verification completed.
   Behavioral assertion check: PASS.

tokenizer-mismatch-module-page-004

1. PASS — focused tests cover route, registry, and English messages together.
2. PASS — coverage asserts discovery expectations.
3. PASS — coverage would fail on canonical route / registry / discovery regression.
4. PASS — coverage is mostly runtime-observable for this slice.
5. PASS — typecheck passed locally.
6. PASS — tests passed locally.
   Behavioral assertion check: PASS.

Website standards:

1. Architecture and State: PASS — the change stays within existing content/registry/runtime patterns.
2. Components and Interaction: PASS — it reuses the canonical module page components instead of introducing one-offs.
3. Styling and Visual Consistency: PASS — it uses the established docs shell and asset renderers.
4. Accessibility: PASS — browser rendering preserved semantic headings and link navigation; no obvious regressions found.
5. Responsive Design: PASS — no responsive-specific regression was evident in the rendered shell.
6. Performance and Resilience: PASS — no new client-side data path or failure mode was introduced.
7. Browser Compatibility and Progressive Enhancement: PASS — the page rendered correctly in the browser pass.
8. Localization: PASS — copy remains page-localized through messages and stays English-only as requested.
9. Testing and Diagnostics: PASS — automated coverage plus browser verification were provided.

Docs writing standards:

1. PASS — the page is understandable in isolation.
2. PASS — the narrative stays focused on the concept.
3. PASS — the first sections explain what it is and why it matters in plain language.
4. PASS — the full name is used before shorthand.
5. PASS — sections have distinct jobs.
6. PASS — the page is not math-heavy enough to require more formal derivation than it has.
7. PASS — the page includes a graph and comparison aid appropriate to the topic.
8. PASS — the math definitions stay symbol-level.
9. PASS — no reader-shortcut or workflow-language callouts appear in customer-facing copy.
10. FAIL — references/citations are not correct for the claims currently made. The GPT-2 report does not justify presenting tokenizer mismatch as a released module authored by that paper's authors, and it does not cover the page's modern chat-template failure-mode framing.
11. PASS — related docs/tags help discovery without carrying the explanation.
12. PASS — the copy is concise and direct.

Graphing standards:

1. PASS — a module flow graph is a reasonable presentation for this failure mode.
2. PASS — the page's formula renders through KaTeX-backed math.
3. PASS — the flow direction is clear.
4. PASS — the graph rendered in browser.
5. PASS — it fits the existing module-page presentation family.
6. PASS — the graph intent is clear.
7. PASS — the chosen presentation is acceptable for the concept.
8. FAIL — I do not see a legend on the graph.
9. PASS — the graph has a visible title/context via the surrounding section.
10. PASS — no obvious color-contrast issue was apparent in the browser pass.

Required fixes before this can merge:

1. Remove or correct the false release/authorship/source attribution for module.tokenizer-mismatch. If this topic does not have a real introducer/source in the project data model, omit those metadata fields instead of forcing GPT-2 onto the page.
2. Bring citations in line with the actual claims made on the page, especially the tokenizer/special-token/chat-template mismatch claims.
3. Rebase and resolve the current PR merge conflict state.

[AndreasAbdi]

Addressed the blocking feedback on the current PR head.

Fixed item 1: removed the false GPT-2 release attribution from module.tokenizer-mismatch by dropping releaseDate, authors, and sourceId, and added an explicit releaseMetadataExceptionReasons entry because this page documents a failure mode rather than a discrete released component. Added runtime assertions that buildPageReleaseMetadata(record) is null for tokenizer mismatch and verified the rebuilt production page no longer renders a Released row or GPT-2 authors in At a Glance.

Fixed item 2: replaced the unsupported GPT-2-only citation set with four references that support the actual claims on the page:

• citation.zero-shot-tokenizer-transfer
• citation.sennrich-bpe
• citation.hugging-face-chat-templates
• citation.hugging-face-chat-templates-docs

Fixed item 3: rebased the branch onto current origin/main, resolved the generated published-docs-registry-ids.ts conflict by keeping the manifest-backed runtime path, regenerated src/lib/content/published-docs-registry-manifest.ts, and force-pushed the rebased branch with lease. gh pr view --json mergeable,mergeStateStatus now reports mergeable: MERGEABLE and mergeStateStatus: UNSTABLE because CI is currently running on the new head.

Local verification on the rebased head:

• make lint
• make typecheck
• make validate-data
• make test
• Production browser verification on http://127.0.0.1:3478/docs/modules/tokenizer-mismatch after a fresh rebuild: title/summary/tags/related links rendered, At a Glance omitted release metadata, and References rendered the new citation set.

[AndreasAbdi]

Addressed the follow-up mergeability failure on the current PR head.

GitHub pull_request CI was running the merge ref against newer origin/main, where sentencepiece had become a published module route and the bundled graph inventory had grown. That made two stale assertions fail only in CI:

• src/lib/content/graph-registry-runtime.test.ts now expects 48 bundled graph records on the rebased head.
• src/lib/content/tokenizer-mismatch-registry.test.ts now treats module.sentencepiece as a published related-doc target with /docs/modules/sentencepiece and one fewer planned-related marker.

I rebased the branch onto current origin/main, committed the narrow test updates on top (ee830abcee840b70ada71385fdd3d862db974f2e), and re-ran:

• bun test src/lib/content/graph-registry-runtime.test.ts src/lib/content/tokenizer-mismatch-registry.test.ts
• make lint
• make typecheck
• make validate-data
• make test

The PR head is now ee830abcee840b70ada71385fdd3d862db974f2e, the diff still contains the tokenizer-mismatch slice plus this mergeability-only follow-up, and a fresh CI run has started on that head.

[AndreasAbdi]

BLOCKING — reviewed PR #123 on head ee830abcee840b70ada71385fdd3d862db974f2e.

Primary findings:

1. gh pr view 123 --json mergeable,mergeStateStatus currently reports mergeable: CONFLICTING and mergeStateStatus: DIRTY on June 19, 2026, so this PR is not mergeable as-is. Please rebase onto current origin/main, resolve the conflicts, and push a new head before asking for merge again.
2. factory/docs/standards/graphing-standards.md:69 requires every graph to have a legend. The new tokenizer-mismatch compute-flow graph in src/content/registry/graphs/tokenizer-mismatch-compute-flow.json:1 renders without any legend in the browser pass, and the page only mounts it directly through src/content/docs/modules/tokenizer-mismatch/page.mdx:40-44. This is still a standards failure on a newly added graph.

Quality checks:

• make test: PASS locally.
• Live required checks on ee830abcee840b70ada71385fdd3d862db974f2e: PASS (lint, typecheck, test, test-verify-contract, coverage, test-build-contract, test-integration, validate-data, linkcheck, build-export, ci).
• Browser verification: PASS on a local production build at http://127.0.0.1:3478/docs/modules/tokenizer-mismatch, http://127.0.0.1:3478/search?q=wrong%20tokenizer, and http://127.0.0.1:3478/tags/tokenization for route rendering, search discoverability, tag discoverability, related links, and removal of the false Released metadata row. Only console issue was favicon.ico 404.
• Citation spot-check: PASS. The current source set supports the claims being made: ZeTT supports tokenizer/model coupling and tokenizer swapping, and Hugging Face chat-template docs explicitly warn that wrong control tokens drastically hurt performance.

Project acceptance criteria:

1. PASS — /docs/modules/tokenizer-mismatch exists and the page explains what tokenizer mismatch is, why it matters, and where it appears in practice.
2. PASS — module.tokenizer-mismatch now has valid tokenizer-family registry metadata without the false GPT-2 release attribution.
3. PASS — the page and registry connect to tokenizers-overview, bpe, wordpiece, sentencepiece, special-tokens, embedding, and gpt-3.
4. PASS — the page is discoverable through search and the tokenization tag landing page for representative mismatch queries.
5. PASS — the slice stays English-only and scoped to the intended tokenizer-mismatch bundle.
6. PASS — local make test passed and the current live CI suite is green.

User story acceptance criteria:
tokenizer-mismatch-module-page-001

1. PASS — the registry record is published with the expected id, slug, module classification, aliases, tags, and related metadata.
2. PASS — classification follows the module-family tokenization path, not a concept/glossary fallback.
3. PASS — registry relationships match the intended tokenizer neighbors plus the representative model link.
4. PASS — the diff stays scoped to this page slice and its narrow runtime/test surfaces.
5. PASS — typecheck passed in live CI.
6. PASS — tests passed locally and in live CI.
   Behavioral assertion check: PASS — registry/discovery behavior is asserted, not just file presence.

tokenizer-mismatch-module-page-002

1. PASS — canonical route, frontmatter, English messages, and local assets are present.
2. PASS — the page opens with one folded summary and uses the standard module shell.
3. PASS — the copy explains wrong vocabulary, splitting rules, special tokens, and prompt-format mismatch.
4. PASS — the copy explains reader-visible consequences such as token-count drift and broken chat boundaries.
5. PASS — the customer-facing copy is English-only and free of process/meta prose.
6. PASS — typecheck passed in live CI.
7. PASS — tests passed locally and in live CI.
8. PASS — browser verification completed.
   Behavioral assertion check: PASS.

tokenizer-mismatch-module-page-003

1. PASS — representative search queries return /docs/modules/tokenizer-mismatch.
2. PASS — related-doc and tag surfaces render the expected nearby pages.
3. PASS — the tokenization tag landing page leads to the canonical page without typing the route directly.
4. PASS — browser rendering shows title, summary, tags, and related links without missing placeholders or broken links.
5. PASS — typecheck passed in live CI.
6. PASS — tests passed locally and in live CI.
7. PASS — browser verification completed.
   Behavioral assertion check: PASS.

tokenizer-mismatch-module-page-004

1. PASS — focused automated coverage ties together route, registry, and English messages.
2. PASS — coverage asserts discovery expectations for the page.
3. PASS — coverage would fail on canonical-route, registry-linkage, or discovery regressions.
4. PASS — coverage is behavior-oriented for this slice and not just inventory scanning.
5. PASS — typecheck passed in live CI.
6. PASS — tests passed locally and in live CI.
   Behavioral assertion check: PASS.

Website standards:

1. PASS — architecture and data ownership stay within existing docs/runtime layers.
2. PASS — the page reuses the canonical module-page components and explicit interfaces.
3. PASS — styling stays within the established docs shell and shared presentation patterns.
4. PASS — browser verification showed no obvious accessibility regression in headings, links, or navigability.
5. PASS — the rendered page remained usable in the browser verification pass.
6. PASS — no new brittle client-side data path or resilience issue was introduced.
7. PASS — the browser-visible route works as a progressively enhanced static/docs surface.
8. PASS — user-facing copy remains localized through page messages and the slice stays English-only as requested.
9. PASS — automated coverage plus browser QA provide appropriate evidence for the change.

Docs writing standards:

1. PASS — the page is understandable in isolation.
2. PASS — the narrative stays focused on the topic and avoids page-meta prose.
3. PASS — the first sections explain both what the topic is and why it matters in plain language.
4. PASS — the full name is used before shorthand.
5. PASS — sections have distinct jobs without obvious repetition.
6. PASS — the page is not math-heavy enough to require more formal derivation than it includes.
7. PASS — the page includes a graph and schema aid appropriate to the concept.
8. PASS — the math definitions stay symbol-only and concise.
9. PASS — no reader-shortcut callouts or workflow language appear in customer-facing copy.
10. PASS — the current citation set is materially aligned with the claims on the page.
11. PASS — related docs and tags support discovery without carrying the explanation.
12. PASS — the prose is concise and direct.

Graphing standards:

1. PASS — a compute-flow graph is a reasonable presentation for this failure mode.
2. PASS — the page’s formula renders through KaTeX-backed math.
3. PASS — the graph presents directional flow clearly.
4. PASS — the graph rendered in the production browser pass.
5. PASS — it fits the existing module-page graph family.
6. PASS — the graph’s intent is understandable from the rendered page.
7. PASS — the chosen graph type is appropriate for the concept.
8. FAIL — no legend is rendered for the new graph, despite the standard requiring one.
9. PASS — the surrounding section gives the graph a visible title/context.
10. PASS — no obvious WCAG color-contrast issue was apparent in browser QA.
    Node graph rule 8: PASS — node labels were legible in the browser pass.
    Node graph rule 9: PASS — edge flow remained visible.
    Node graph rule 10: PASS — the graph uses the established themed node styling.
    Node graph rule 10.1: PASS — this graph is not a baseline-vs-variant comparison, so no differential highlight was required.
    Comparison-chart rule 1: PASS — not applicable; this graph is not a comparison graph.
    Comparison-chart rule 2: PASS — not applicable; this graph is not a comparison graph.

Required fixes before merge:

1. Rebase onto current origin/main, resolve the current merge conflict state, and push a fresh head so GitHub reports the PR as mergeable.
2. Add a visible legend for the tokenizer-mismatch compute-flow graph so the new graph conforms to factory/docs/standards/graphing-standards.md.