feat: classify onboarding-generated topics into existing categories

[byteclimber]

Summary

Onboarding-generated prompts previously arrived under the dimension-root
category tag with no sub-category assignment: category is an open,
customer-authored dimension (unlike intent/type, which have fixed
taxonomies), so there was no existing vocabulary to classify against. This
left every onboarding prompt uncategorized until a human manually re-tagged
it — see adobe/serenity-docs#44.

This PR adds server-side AI classification of onboarding topics against the
brand's existing top-level categories, per the implementation plan in
adobe/serenity-docs#479.

Changes

• category-taxonomy.js (new): system-prompt builder and response
  validator for topic→category classification. Requires an explicit
  NO_MATCH sentinel rather than forcing a pick, and rejects any category
  name outside the candidate list as a hallucination (invalid_value)
  rather than silently accepting it. Confidence floor is 0.7 (stricter
  than intent classification's 0.5), since a wrong category mints a
  durable, customer-visible sub-category tag.
• category-classification.js (new): classifyTopicCategories,
  mirroring classifyPromptIntents's fail-open ladder (budget skip-gate,
  Azure-not-configured fallback), simplified for the small per-run topic
  count (≤5 topics, single classify pass, no retry).
• markets-subworkspace.js: generateAndAttachPrompts now preserves
  topic→prompts grouping instead of flattening immediately, fetches the
  brand's existing category children, classifies each topic, and
  creates/resolves a matched topic as a sub-category tag. Every step is
  best-effort and fails open: reading existing categories, classifying, and
  creating each sub-category tag all fall back to leaving affected topics
  uncategorized without blocking the prompt write.

Safety / backward compatibility

• When Azure OpenAI isn't configured (the default in all current
  environments/tests), classification is skipped and prompts are written
  exactly as before — the prompt tag-bucketing key extension
  (type\0intent\0categoryTagId, with categoryTagId defaulting to '')
  is byte-identical to the old key whenever no category is resolved.
• All 111 pre-existing markets-subworkspace tests pass unmodified; 2 new
  integration tests cover the happy-path match+tag-creation and the
  fail-open path when sub-category tag creation fails.
• Full repo test suite (18,889 tests) passes; tsc type-check and ESLint
  are clean on all touched/new files.

Out of scope (tracked as follow-ups per the plan)

• Confidence-threshold calibration against real labeled data (plan §5.3).
• Backfill of already-onboarded, currently-uncategorized prompts (plan §6).
• Multi-category ambiguity handling (current design assumes single-best-match).

Refs: adobe/serenity-docs#44, adobe/serenity-docs#479

Change Management

cm-assessment: v1
changeType: standard
impact: unnoticeable
risk: minor
scope: single-repo
relatedPRs: ["adobe/serenity-docs#479"]
rationale: "Adds AI classification of onboarding topics into existing brand categories, wired into the onboarding prompt-generation write path. Fails open at every layer (budget/config skip, classification failure, or sub-category-creation failure each leave affected topics uncategorized without blocking the write) and is byte-identical to prior behavior whenever Azure OpenAI is not configured, which is every current environment/test default. No destructive or irreversible data operations, no auth/security-boundary changes, no external API contract changes. A misclassification only creates an extra sub-category tag, which is correctable after the fact."
recommendations: "Monitor the CategoryOutcome EMF metric after rollout to check hallucination/no-match/low-confidence rates before considering a lower confidence floor; run a labeled-sample calibration pass per the plan before broad reliance on the classifier's output."
backout: "Redeploy the previous release; the change is fail-open so reverting only removes future category classification, it does not need to unwind existing tags."

[codecov]

Codecov Report

❌ Patch coverage is 98.64865% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../support/serenity/handlers/markets-subworkspace.js	96.96%	4 Missing ⚠️
src/support/serenity/category-classification.js	98.76%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!