feat: add Azure embedding client and semantic index utils (LLMO-7445)

docs/adr/0002-embedding-client-and-semantic-index-utils.md

@@ -0,0 +1,60 @@
+# ADR-0002: Embedding client placement and semantic-index utils ownership
+
+- **Status:** Accepted
+- **Ticket:** LLMO-7445
+- **Deciders:** Lookup Service working group (LLMO-7445)
+- **Date:** 2026-09-17
+- **Relates to:** PR #1933 (this change) · audit-worker ADR 006 (Lookup Service write foundation) + ADR 007 (topic dimension) · spacecat-api-service#3298 (by-topics read endpoint) · mysticat-data-service semantic tables + RPCs
+
+## Scope
+
+This ADR governs, within `spacecat-shared`, **where embeddings support lives and what the shared layer owns** for the Lookup Service semantic (by-topic) matching: the Azure embeddings client (`@adobe/spacecat-shared-gpt-client`) and the semantic-index storage/retrieval helpers (`@adobe/spacecat-shared-data-access`). It does **not** decide the product/design of semantic matching itself — that is the cross-repo Lookup Service design (audit-worker ADR 006/007) — only the shared-library boundaries the same way ADR-0001 did for the Semrush client.
+
+## Context
+
+Semantic by-topic matching needs one embedding model used on both sides — the **write** path (audit-worker embeds opportunity topic titles) and the **read** path (api-service embeds the query text on a cache miss) — so their vectors are comparable by construction. Before this change `spacecat-shared` had no embeddings client (`gpt-client` was chat/completions only) and no vector storage helpers.
+
+Two placement questions had to be settled so future consumers don't re-litigate them:
+
+1. Does the embeddings client belong in `gpt-client`, or in a new `spacecat-shared-embedding-client` package?
+2. What does the shared layer own vs. what does each consumer own?
+
+## Decision
+
+### 1. The embeddings client extends `gpt-client`; it is not a new package
+
+`AzureEmbeddingClient` ships in `@adobe/spacecat-shared-gpt-client` alongside `AzureOpenAIClient`, because embeddings are the **same vendor and auth** (Azure OpenAI endpoint/key/api-version) as the chat client — that package already groups multiple Azure/LLM-provider clients. A separate package would duplicate the auth/transport shape for no isolation benefit. It is a **distinct class**, not an overload of the chat client (embeddings ≠ chat/completions).
+
+### 2. Consumers depend on the `EmbeddingProvider` interface, not the concrete client
+
+The client implements a minimal `EmbeddingProvider` typedef (`createEmbeddings(inputs, options?) => number[][]`). Consumers (and the api-service engine) type against the interface, so the provider/model can be swapped later — a coordinated re-embed, no code-shape change. A future extraction into a dedicated package stays non-breaking behind this seam.
+
+### 3. Ownership boundary — the shared client owns transport
+
+Following ADR-0001's test (*true for every consumer → shared; specific to one → consumer*):
+
+**Shared client (`AzureEmbeddingClient`) owns:**
+- Authentication (Azure `api-key`), endpoint/version resolution (`AZURE_EMBEDDING_*` with fallback to `AZURE_OPENAI_*`).
+- Retry/backoff for transient 429/5xx (honoring a **bounded** `Retry-After`, exponential backoff with jitter, capped).
+- Batching a set of inputs into one call and returning vectors **in input order**, with a response-length guard.
+
+**Consumers own:**
+- Caching. The durable query-embedding cache (`semantic_query_embedding`) and its key policy (`hash(normalizedText)+model+dims`) live in the data-access helpers/consumer, not the client — staleness/keying is a consumer concern.
+- Model/dimension policy at the call site (the cache key and the stored `vector(N)` column pin the model+dims contract).
+- Translating a client failure into an HTTP response (api-service returns 503; the audit-worker write path degrades best-effort).
+
+### 4. Semantic-index helpers live in `data-access`, mirroring the URL index
+
+`semantic-index.utils.js` mirrors `url-index.utils.js`: `syncOpportunitySemantic` (full-replace per `source_type`), `lookupOpportunitiesByVector` (ANN read RPC wrapper), the `semantic_query_embedding` cache helpers, and `copyEntityVectors` (a thin wrapper over the `wrpc_copy_opportunity_semantic_vectors` **server-side** write RPC — the copy runs as `INSERT … SELECT`, no rows materialized in the caller). Text normalization + hashing live in this layer so a value written always matches a later read/re-sync. **The shared layer stores and reads pre-embedded vectors; it never embeds** — embedding is the writer's (audit-worker's) job via the client above.
+
+## Consequences
+
+- `gpt-client` gains an embeddings surface and its own `AZURE_EMBEDDING_*` config (documented in its README); the intended cost, not a drawback.
+- The shared embedding model is a cross-repo contract: both the write and read paths must point at the **same deployment**. A model change is a coordinated re-embed of the opportunity index; the query cache self-invalidates via its `model+dims` key.
+- Consumers stay thin: they inject the `EmbeddingProvider`, own their cache/status mapping, and call the `data-access` helpers — no matching logic is reimplemented per consumer.
+
+## Alternatives considered
+
+- **A new `spacecat-shared-embedding-client` package.** Rejected for now: same vendor/auth as `gpt-client`; deferred behind the `EmbeddingProvider` seam so a later extraction is non-breaking.
+- **Overloading `AzureOpenAIClient`.** Rejected: embeddings are a different API surface; a distinct class keeps each client cohesive.
+- **Embedding inside the shared data-access layer.** Rejected: transport/auth belongs in a client (ADR-0001 boundary), and the write side (audit-worker) owns when/what to embed.

packages/spacecat-shared-data-access/src/util/index.js

@@ -36,6 +36,26 @@ export {
   lookupEntityIdsByUrl,
 } from './url-index.utils.js';
 
+export {
+  SEMANTIC_INDEX_TABLES,
+  QUERY_EMBEDDING_TABLE,
+  SEMANTIC_SEARCH_RPC,
+  COPY_VECTORS_RPC,
+  SEMANTIC_CHUNK_SIZE,
+  MAX_SOURCE_TEXT_LENGTH,
+  normalizeText,
+  hashText,
+  cleanTopicText,
+  serializeVector,
+  parseVector,
+  syncOpportunitySemantic,
+  copyEntityVectors,
+  lookupOpportunitiesByVector,
+  getQueryEmbedding,
+  upsertQueryEmbedding,
+  touchQueryEmbedding,
+} from './semantic-index.utils.js';
+
 /**
  * Datastore types that collections can use to declare their storage backend.
  * @readonly