fix(serenity): scope create-path dedup to input via search, add fixed-walk fallback by dzehnder · Pull Request #3304 · adobe/spacecat-api-service

dzehnder · 2026-09-17T09:25:59Z

1. Abstract

Fixes adding prompts failing (single "Add prompt" and CSV import) for large-corpus brands by scoping the create/upsert dedup lookup to the incoming texts instead of walking the whole Semrush project corpus on every write. Ships behind a default-ON kill-switch whose OFF path is a bounded, fully-capped corpus walk (never the pre-fix serial walk).

2. Reasoning

handleCreatePrompts (and its subworkspace twin) built a text -> existing prompt dedup index by walking the entire project corpus via buildExistingPromptIndex — up to 20 sequential 1,000-item POST /aio/prompts/by_tags calls — before any prompt is written. The cost scales with the number of existing prompts, not the input, so on Adobe Helpx (~38,764 prompts) the walk hits the page cap every time and the ~20 serial upstream calls exceed the ~15s Fastly edge budget → the request fails whether the customer adds 1 prompt or 16. Third instance of the serial-corpus-walk class in this incident (read #3285 merged, delete #3288, now create); RCA #3283/#3287, customer thread since 2026-09-16.

The dedup index exists to decide upsert-vs-create per input: an already-present text must have its tags replaced via the existing prompt's id, not go down the create path — where the upstream create folds a repeated text into existing_count but still attaches the given tags, silently stacking a second tag on the live prompt (the additive-tag bug #3219/LLMO-7422, "could never be undone").

3. High-level overview of the changes

New buildTargetedPromptIndex (default, kill-switch SERENITY_TARGETED_CREATE_LOOKUP ON): one search call per distinct (projectId, text), exact-match only (name.trim() === text.trim()), building the same {byText, byLower} index. Cost scales with input, not corpus (1 add = 1 call, 16-row CSV = 16 calls), and it reduces the /aio/prompts/by_tags load Semrush flagged for small adds. Phase 0 verified live (2026-09-17) that upstream search is literal substring matching, an exact text returns the row at position 0 (total=1) across unicode/punctuation, and metacharacters are literal (no operator injection).
buildExistingPromptIndex (kill-switch OFF fallback): reworked from a serial for loop to a bounded-concurrency page fetch (BULK_CREATE_CONCURRENCY = 8), and MAX_PROMPT_INDEX_PAGES raised 20 → 100 (100k prompts) so a large brand is fully indexed. An incomplete index is a correctness bug (beyond-cap rows resolve as "new" and tag-stack), not just a perf one; the biggest real corpus is ~76k (Prompt Library: bulk delete issues one round-trip per prompt id, and the list read materializes the whole collection — both 503 at the Fastly edge for large brands #3279). Flag read per-request; unset/any non-'false' = targeted (default ON), explicit 'false' = walk.
Whole-project degradation: any lookup failure (a search error, a full page with no exact hit, or a budget timeout) degrades that whole project to an index error → its inputs fail itemized (HTTP 200), never a silent "new"/tag-stack. isIndexError is per-project; no per-input isolation.
Deadline threaded into the index build + fan-out: writeDeadline was only on classifyPromptIntents; the index build and create fan-out ran ungated. Now once the budget is spent, remaining inputs fail itemized (503) so the request returns a 2xx partial before the edge terminates the Lambda mid-flush.
Write-path observability (in this PR): one structured end-of-request log.info on both publish branches — {brandId/workspaceId, created, updated, skipped, failed, published, upstreamCallCount, elapsedMs, targetedLookup}. Serenity writes are otherwise invisible in cdn_prod and silent in backend_prod (why this incident was code-only-diagnosable); this line is the rollback detector and validation evidence.
Covers both handleCreatePrompts (flat) and handleCreatePromptsSubworkspace at the shared buildPromptIndexByProject.

4. Required information

Jira / issue: LLMO-7615
Related issues: Prompt Library: faceted-v1 search walks the entire Semrush project corpus serially, 503s at the Fastly edge for large brands (distinct from #3279/PR #3280) #3283 (read-path RCA), Prompt Library: serenity bulk-delete 503s at the edge while the Lambda keeps running (likely succeeds server-side) — make it async #3287 (delete-path RCA)
Related PRs: fix(serenity): cap faceted prompt corpus walk to fail fast on large brands #3285 (read-path fix, merged), feat(serenity): async bulk-delete to stop edge-timeout 503s on large brands #3288 (delete-path async, in review); frontend adobe/project-elmo-ui#3185
Spec: mysticat-workspace/local/serenity-prompts-create-path-corpus-walk-fix-spec.md (5-persona architecture review + live Phase 0 verification 2026-09-17)

7. Additional information outside the code

The kill-switch is default ON — Option A (targeted lookup) is the shipped behavior, not gated off. Turning the flag OFF lands on the fixed bounded/capped walk (a correct, fast state), never the pre-fix serial-20-page walk that caused the incident. Rollout can be per-brand via env allowlisting if a canary is wanted; the flag is a temporary kill-switch (add a removal ticket once soaked). search load caveat: A reduces by_tags load for the common small-add case, but a max 500-item import is up to 500 light search calls vs the walk's ~40 — quantify by regime for the Semrush conversation rather than claiming a one-directional win.

8. Test plan

(a) Local unit tests (prompts.test.js + prompts-subworkspace.test.js, both handlers): existing text → upsert (same semrushPromptId, tags replaced not stacked); new text → create; cost assertion — targeted call count == distinct (projectId, text) pairs; exact-match rejects a same-project substring over-match; a search failure degrades the whole project (inputs fail itemized, never treated as new); a full page with no exact hit degrades the project; write-budget-exhausted marks inputs failed (503); fixed-walk (flag off) pages concurrently and recognises a page-2 prompt in one batch. 260 passing across the two suites; 736 passing across all serenity handler suites; npm run type-check and eslint clean.
(b) Per-env (recommended before enabling widely): seed a large synthetic corpus on a dev brand, add a single prompt + import the 16-row sample-run-for-acrobat-prompt-upload.csv + a full 500-item request; assert 2xx within the edge budget and that a known-existing text upserts (same id, tags replaced not stacked). Prod: a flag-gated canary on Helpx with a disposable prompt, id+tags checked before/after, then cleanup.

Change Management

cm-assessment: v1
changeType: standard
impact: unnoticeable
risk: minor
scope: single-repo
relatedPRs: ["adobe/spacecat-api-service#3285", "adobe/spacecat-api-service#3288"]
rationale: "Reshapes an internal create/upsert dedup read from a full-corpus walk to a per-input search lookup behind a default-ON kill-switch; the response contract is unchanged, the dangerous failure direction (treating an existing prompt as new -> silent tag-stack) is closed by whole-project degradation, and disabling the flag lands on a corrected bounded/capped walk, never the pre-fix serial walk."
recommendations: "Enable on a dev brand with a seeded large corpus first; canary on Helpx via a flag-gated per-brand allowlist checking id+tags before/after; watch the new create-prompts completed log (upstreamCallCount, elapsedMs, targetedLookup) and the Semrush by_tags load."
backout: "Set SERENITY_TARGETED_CREATE_LOOKUP=false (instant revert to the corrected bounded walk) or redeploy the previous release."

🤖 Generated with Claude Code

…-walk fallback The create/upsert path built its existing-prompt dedup index by walking the whole Semrush project corpus (buildExistingPromptIndex: up to 20 serial 1000-item by_tags calls) before any write. On a large brand (Adobe Helpx, ~38.7k prompts) that blows the ~15s Fastly edge budget, so adding prompts (single add and CSV import) fails regardless of input size. Third instance of the serial-corpus-walk class (read #3285, delete #3288, now create). Default (kill-switch SERENITY_TARGETED_CREATE_LOOKUP ON): buildTargetedPromptIndex looks up ONLY the incoming texts via one `search` call per distinct (projectId, text), exact-match only — cost scales with input, not corpus. Phase 0 verified live (2026-09-17) that upstream `search` is literal substring matching, exact text returns the row at pos 0, metacharacters literal. Kill-switch OFF falls back to the walk, now bounded-concurrency and capped at 100k (was 20k) so a large brand is fully indexed — never the pre-fix serial walk. Failure/ambiguity degrades the whole PROJECT to an index error (inputs fail itemized, HTTP 200), never a false "new" (which would silently tag-stack an existing prompt via the create endpoint's existing_count+tag-attach behavior). writeDeadline is now threaded into the index build and fan-out, returning a 2xx partial before the edge kill. A structured end-of-request log (upstreamCallCount, elapsedMs, targetedLookup) makes create/upsert diagnosable from logs. Covers both flat and subworkspace create handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

codecov · 2026-09-17T09:30:47Z

Codecov Report

❌ Patch coverage is 96.98492% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../support/serenity/handlers/prompts-subworkspace.js	72.09%	12 Missing ⚠️

📢 Thoughts on this report? Let us know!

@PARAM

…lose review gaps Addresses the PR #3304 review findings: - Thread writeDeadline into buildTargetedPromptIndex/buildExistingPromptIndex so a large single-project add short-circuits mid-build (degrades to itemized 503) instead of running the edge budget out before the fan-out. - Guard an exact-name match with no usable id: degrade the project rather than silently falling through to create-as-new (tag-stack). - Make the full-page/falsy-id degrade client message redaction-safe (generic 'existing-prompt lookup unavailable'); keep specifics in the log. - Update stale @PARAM docs for env/writeDeadline on both create handlers; add the vendor-search residual-contract note, the short-text false-degrade note, the flag-removal note, the .env.example entry, and the batch-short-page doc fix. - Tests: per-item deadline guard (guard 2), build-phase deadline guard (both build fns), falsy-id degrade, 20->100 cap recognition + cap-hit warn, duplicate-text cost dedup, redacted-message assertions; subworkspace parity for full-page, falsy-id, deadline, and flag-off walk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-17T11:42:20Z

This PR will trigger a patch release when merged.

MysticatBot

Hey @dzehnder,

Verdict: Request changes - two issues to address before merge.
Complexity: HIGH - large diff; API handler changes.
Changes: Scopes create-path dedup lookup to incoming texts via targeted search instead of walking the whole corpus, with a bounded-concurrency walk fallback behind a kill-switch (5 files).
Note: CI build and it-postgres checks are still pending - resolve before merge.

Must fix before merge

[Important] Case-insensitive dedup regression in targeted lookup - the strict === exact-match misses case variants that the walk path catches via byLower - src/support/serenity/handlers/prompts.js:1960 (details inline)
[Important] New env var SERENITY_TARGETED_CREATE_LOOKUP not documented in docs/serenity.md operator guide - .env.example:96 (details inline)

Non-blocking (3): minor issues and suggestions

nit: if (exact && exact.id) is redundant after the preceding if (exact && !exact.id) throw - a plain if (exact) suffices - src/support/serenity/handlers/prompts.js:1978
suggestion: TARGETED_LOOKUP_LIMIT = 25 may be low for short/common texts with many substring siblings - consider raising to 50 or making it env-configurable to reduce false-degrades without a deploy - src/support/serenity/handlers/prompts.js:1754
nit: The kill-switch is marked temporary but no removal ticket is linked - create one now so it does not calcify - .env.example:96

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 15s | Cost: $12.87 | Commit: 37aac9952cc430c4e42092b160f57315d76af03a
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

MysticatBot · 2026-09-17T12:02:55Z

+    }
+    onUpstreamCall?.();
+    const resp = await transport.listPromptsByTags(semrushWorkspaceId, projectId, {
+      tag_ids: [], search: text, page: 1, limit: TARGETED_LOOKUP_LIMIT,


issue (blocking): The === exact-match check is case-sensitive, but the corpus walk path (buildExistingPromptIndex) catches case variants via byLower - and the code comments at line ~1797 explain exactly why: "case-sensitive where upstream's own dedupe is not would send a case variant back down the create path and straight into the additive attach."

If a stored prompt is "Best Shoes" and the input is "best shoes", upstream search (likely case-insensitive substring) returns the row, but === misses it. On a non-full page the function concludes "not found", the input takes the create path, and upstream's case-insensitive dedupe silently stacks a second tag - the exact bug this PR fixes on the corpus-size axis.

Fix: Add a case-insensitive fallback after the strict check:

const exact = items.find((it) => String(it?.name ?? '').trim() === text) ?? items.find((it) => String(it?.name ?? '').trim().toLowerCase() === text.toLowerCase());

When the fallback matches, populate both byText (keyed on the input text) and byLower as normal. The full-page guard still applies if neither match hits on a full page.

Fixed in b70c5d4. buildTargetedPromptIndex now matches case-insensitively (name.trim().toLowerCase() === text.trim().toLowerCase()) and keys byText/byLower by the matched stored name, mirroring the walk path's dedup so a case variant (stored "Best Shoes" vs input "best shoes") upserts instead of falling through to create-as-new. Added a case-variance regression test to both the flat and subworkspace suites.

MysticatBot · 2026-09-17T12:02:55Z

 # Generation model sent to DRS (canonical contract default; override per env).
 DRS_PROMPT_GENERATION_MODEL=gpt-5-nano
 SERENITY_ASYNC_PROMPT_GEN=false
+# Create-path dedup kill-switch. DEFAULT ON (unset, or anything but 'false', uses


issue (blocking): This kill-switch controls production dedup behavior and is called out for per-brand canary rollouts (PR description section 7), but the operator guide (docs/serenity.md, referenced as "the Serenity operator guide" in the repo's CLAUDE.md and README) has no entry for it. Operators troubleshooting a canary or rollback need to find this flag in the docs, not in code comments.

Fix: Add a row to the environment configuration table in docs/serenity.md documenting SERENITY_TARGETED_CREATE_LOOKUP, its default, its effect, and that it is temporary.

Fixed in b70c5d4. Added a SERENITY_TARGETED_CREATE_LOOKUP entry to the Serenity operator guide (docs/serenity.md, Environment configuration): default ON, 'false' falls back to the corrected bounded/capped walk (never the pre-fix serial walk), per-brand canary/rollback lever, and the targetedLookup/upstreamCallCount signals to watch.

…witch Address MysticatBot review on #3304: - buildTargetedPromptIndex now matches existing prompts case-insensitively (keyed by stored name into byText/byLower), mirroring the walk path's dedup so a case variant (stored "Best Shoes" vs input "best shoes") upserts instead of falling through to create-as-new and silently tag-stacking. - Document SERENITY_TARGETED_CREATE_LOOKUP in the Serenity operator guide (docs/serenity.md): default ON, 'false' fallback to the corrected walk, per-brand canary/rollback, and the targetedLookup/upstreamCallCount signals. - Add case-variance regression tests to both flat and subworkspace suites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

MysticatBot

Mysticat could not complete a verified review of this pull request.

This is a service-side gap, not a finding about your code: no binding APPROVE or REQUEST_CHANGES review could be produced and verified for the current commit.

Current head: b70c5d47e7742b4f4783d655a7862ca51bda21d3
Failure category: claude_crash
Request ID: 2e35c6e0-b29a-11f1-8bdc-dc491fde0d68

Please re-request a review (e.g. push a new commit or re-request from the configured reviewer) to try again.

…ath-corpus-walk

MysticatBot

Mysticat could not complete a verified review of this pull request.

This is a service-side gap, not a finding about your code: no binding APPROVE or REQUEST_CHANGES review could be produced and verified for the current commit.

Current head: 95329c00ce634f1b36bd998aa39777cac52f1449
Failure category: claude_crash
Request ID: 58cd2370-b29b-11f1-87b5-480a70a2afa0

Please re-request a review (e.g. push a new commit or re-request from the configured reviewer) to try again.

MysticatBot

Mysticat could not complete a verified review of this pull request.

This is a service-side gap, not a finding about your code: no binding APPROVE or REQUEST_CHANGES review could be produced and verified for the current commit.

Current head: 95329c00ce634f1b36bd998aa39777cac52f1449
Failure category: claude_crash
Request ID: a6f17e50-b29d-11f1-8e42-bc25acb5e06a

Please re-request a review (e.g. push a new commit or re-request from the configured reviewer) to try again.

review failing

# [1.823.0](v1.822.0...v1.823.0) (2026-09-17) ### Bug Fixes * **serenity:** scope create-path dedup to input via search, add fixed-walk fallback ([#3304](#3304)) ([a7e2ee4](a7e2ee4)), closes [#3285](#3285) [#3288](#3288) [3283/#3287](#3287) [Hi#level](https://github.com/Hi/issues/level) [#3279](#3279) [#3283](#3283) [#3287](#3287) [#3285](#3285) [#3288](#3288) [adobe/project-elmo-ui#3185](https://github.com/adobe/project-elmo-ui/issues/3185) ### Features * GET /sites/:siteId/deployed-opportunities (deployed-opportunities timeline) | LLMO-7669 ([#3299](#3299)) ([8b26421](8b26421)), closes [#2501](#2501)

solaris007 · 2026-09-17T14:18:32Z

🎉 This PR is included in version 1.823.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

dzehnder deployed to dev-branches September 17, 2026 09:31 — with GitHub Actions Active

dzehnder requested a review from MysticatBot September 17, 2026 11:47

dzehnder deployed to dev-branches September 17, 2026 11:58 — with GitHub Actions Active

MysticatBot requested changes Sep 17, 2026

View reviewed changes

MysticatBot added ai-reviewed Reviewed by AI complexity:high AI-assessed PR complexity: HIGH labels Sep 17, 2026

dzehnder requested a review from MysticatBot September 17, 2026 13:17

MysticatBot requested changes Sep 17, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into fix/serenity-create-p…

95329c0

…ath-corpus-walk

dzehnder requested a review from MysticatBot September 17, 2026 13:26

MysticatBot requested changes Sep 17, 2026

View reviewed changes

dzehnder deployed to dev-branches September 17, 2026 13:31 — with GitHub Actions Active

byteclimber requested review from MysticatBot and byteclimber September 17, 2026 13:42

byteclimber approved these changes Sep 17, 2026

View reviewed changes

MysticatBot previously requested changes Sep 17, 2026

View reviewed changes

dzehnder merged commit a7e2ee4 into main Sep 17, 2026
24 of 25 checks passed

dzehnder deleted the fix/serenity-create-path-corpus-walk branch September 17, 2026 13:45

solaris007 added the released label Sep 17, 2026

Navigation Menu

fix(serenity): scope create-path dedup to input via search, add fixed-walk fallback - #3304

fix(serenity): scope create-path dedup to input via search, add fixed-walk fallback#3304
dzehnder merged 4 commits into
mainfrom
fix/serenity-create-path-corpus-walk

dzehnder commented Sep 17, 2026

Uh oh!

codecov Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

MysticatBot left a comment •

edited

Loading

Uh oh!

MysticatBot Sep 17, 2026

Uh oh!

dzehnder Sep 17, 2026

Uh oh!

MysticatBot Sep 17, 2026

Uh oh!

dzehnder Sep 17, 2026

Uh oh!

MysticatBot left a comment

Uh oh!

MysticatBot left a comment

Uh oh!

MysticatBot left a comment

Uh oh!

Uh oh!

solaris007 commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dzehnder commented Sep 17, 2026

1. Abstract

2. Reasoning

3. High-level overview of the changes

4. Required information

7. Additional information outside the code

8. Test plan

Change Management

Uh oh!

codecov Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

MysticatBot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Must fix before merge

Uh oh!

MysticatBot Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

dzehnder Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

MysticatBot Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

dzehnder Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

MysticatBot left a comment

Choose a reason for hiding this comment

Uh oh!

MysticatBot left a comment

Choose a reason for hiding this comment

Uh oh!

MysticatBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

solaris007 commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Sep 17, 2026 •

edited

Loading

MysticatBot left a comment •

edited

Loading