Signora

Signora is a Voice of Customer dataset build pipeline and generated dataset package for customer-feedback retrieval, product-identity cleanup, and RAG evaluation experiments.

Google Drive Source

Google Drive link: Voice of Customer source folder

Use the Drive folder for source handoff files that should not live directly in git, such as raw archives, source documents, or replacement CSV exports.

Project Journal

Build journal: BUILD_LOG.md

The journal tracks implementation decisions, commands run, retrieval experiments, and observations from testing Signora.

What Is Included

• scripts/build_voc_dataset.py builds the dataset from the provided source inputs.
• scripts/build_retrieval_index.py builds a local semantic index from feedback atoms.
• scripts/query_retrieval.py runs local semantic retrieval with keyword score fusion and metadata filters.
• scripts/tests/test_dataset_integrity.py validates generated counts, public metadata cleanup, and index-exclusion rules.
• scripts/tests/test_retrieval_index.py validates generated retrieval index artifacts.
• dataset/data/public/normalized/feedback_items.jsonl contains accepted public Reddit feedback records.
• dataset/data/public/quarantined/feedback_items.jsonl contains records held out by validation rules.
• dataset/data/processed/feedback_items.jsonl and dataset/data/processed/feedback_atoms.jsonl contain the combined retrieval-ready corpus.
• dataset/reports/ contains audit, quality, and final dataset reports.

Public Reddit Metadata Cleanup

The public Reddit normalization keeps raw CSV inputs unchanged and fixes product identity in generated artifacts:

• Amazon Echo records are normalized under Amazon Echo / Alexa with product_family and product_model.
• Sony WH-1000X records are separated into XM4, XM5, XM6, and series-level identities.
• Reddit thread fields include thread_id, root_post_id, parent_id, and is_root_post where they can be derived conservatively.
• Wyze records with generic subreddit URLs keep unknown thread relationships as null.

Rebuild

From the repository root:

python scripts\build_voc_dataset.py

The build writes regenerated dataset artifacts, reports, manifests, and reference files under dataset/.

Validate

python scripts\tests\test_dataset_integrity.py

Expected result:

dataset integrity checks passed

Local Semantic Retrieval

Install the retrieval dependencies with Python 3.12:

py -3.12 -m pip install -r requirements.txt

Build the production embedding index. The default profile is openai_small, which uses text-embedding-3-small because it was the practical winner in the benchmark: it led the core retrieval metrics while being cheaper than text-embedding-3-large.

py -3.12 scripts\build_retrieval_index.py

For a fully local/offline baseline, build with MPNet instead:

py -3.12 scripts\build_retrieval_index.py --profile mpnet

The generated semantic index is written to dataset/index/<profile>/. It is intentionally ignored by git because it is rebuildable and model-dependent.

Run a retrieval query:

py -3.12 scripts\query_retrieval.py "Why are enterprise administrators struggling with onboarding?" --top-k 5

The production query command now defaults to the recall-oriented pass. It combines semantic, keyword, and metadata/topic candidate routes before reranking and deduplicating the evidence sent to answer generation. Use --mode pipeline to compare against the previous hybrid baseline:

py -3.12 scripts\query_retrieval.py "Why are enterprise administrators struggling with onboarding?" --mode pipeline --top-k 5

Optional filters:

py -3.12 scripts\query_retrieval.py "Which customers complain about delayed notifications?" --product-area notifications --top-k 5

Generate a grounded answer from the recall-oriented evidence:

py -3.12 scripts\answer_query.py "Why are enterprise administrators struggling with onboarding?"

Run the HTTP API locally:

py -3.12 -m uvicorn signora_api:app --app-dir scripts --host 127.0.0.1 --port 8000

The API exposes GET /health, GET /ready, interactive documentation at /docs, and POST /v1/answers. Answer responses are compact by default; set include_evidence to true when the caller also needs the retrieved rows. The root URL opens the analyst interface for questions, filters, uncertainty, retrieval confidence, and citation inspection.

Run the deterministic desktop and mobile analyst-workspace checks with:

npm install
npx playwright install chromium
npm run test:e2e

Answer generation uses the calibrated abstention threshold in config/retrieval_policy.json. Questions below that threshold stop before the generation call. Answered responses use structured output, require inline atom citations, and fail closed if the model cites evidence that retrieval did not return. The default generation model is gpt-5.4-mini; override it with SIGNORA_ANSWER_MODEL or --model.

Run the held-out answer-quality evaluation after changing retrieval, generation prompts, answer models, or citation handling:

py -3.12 scripts\benchmark\evaluate_answers.py --answerable-limit 6 --abstention-limit 6 --fail-on-gate

This keeps answer correctness, abstention, citation predicate precision, groundedness, uncertainty, counterevidence, recommendation separation, and source diversity visible as separate measurements.

The expanded 12-answerable/10-abstention run passed every quality floor. See dataset/reports/answer_quality_expanded.md for the versioned summary.

The query pipeline uses the same model recorded in the index manifest and fuses dense semantic similarity with keyword overlap. The benchmark showed keyword fusion improved retrieval ranking for both local models, so the production default keeps fusion enabled with --semantic-weight 0.75.

Repeatable Benchmark Habit

Use the v2 benchmark whenever the corpus, document recipe, embedding model, filters, or ranking logic changes:

py -3.12 scripts\benchmark\run_reproducible_eval.py --models openai_small mpnet

That command rebuilds model-specific indexes, evaluates dense retrieval and the production-like keyword-fusion pipeline, scores abstention, and regenerates the benchmark report under dataset/benchmark_results/.

To compare the recall-oriented retrieval pass against the production-like hybrid baseline, include the recall mode:

py -3.12 scripts\benchmark\run_reproducible_eval.py --models openai_small --modes pipeline recall
py -3.12 scripts\benchmark\quality_gate.py --model openai_small

The recall mode keeps the same explicit metadata filters, gathers a broader candidate pool using query overlap with structured topic metadata and search text, then reranks before the answer-generation step.

The quality gate requires at least 0.95 predicate coverage and 0.95 abstention balanced accuracy, with no precision or canonical-recall regression against the hybrid baseline.

Current interpretation from the completed experiment:

• text-embedding-3-small is the production default for this benchmark.
• all-mpnet-base-v2 is the strongest local baseline.
• Keyword fusion improved retrieval ranking for both local models.
• MPNet dense retrieval produced the strongest held-out local abstention result.

The lower-level benchmark runner uses 59 predicate-backed retrieval questions and 20 no-answer cases. It reports exact canonical citation recall separately from predicate-backed evidence coverage so broad semantic retrieval questions are not graded only against one small set of pre-selected atom IDs:

py -3.12 scripts\evaluation\run_benchmark.py

Build one isolated index per model first, for example py -3.12 scripts\embeddings\embed_minilm.py; indexes live under dataset/index/<model-slug>/ and are intentionally ignored by git.

Suggested Next Decisions

• Parent-thread context is deferred until ingestion provides stable parent IDs and a threaded-source evaluation set. See dataset/reports/thread_context_readiness.md.
• Build an analyst-facing interface on top of the validated /v1/answers API.

Notes

• The generated CloudDesk corpus is synthetic.
• Public Reddit CSV samples are user-provided source material and should be reviewed before publication.
• Evaluation answer keys and hidden patterns are excluded from the normal vector-index include manifest.