Skip to content

P2-A-v3-evaluator-safety-and-no-go - #1317

Merged
XingYu-Zhong merged 23 commits into
KunAgent:developfrom
SunwardL:codex/p2a-semantic-evaluation-v3
Sep 14, 2026
Merged

XingYu-Zhong merged 23 commits into
KunAgent:developfrom
SunwardL:codex/p2a-semantic-evaluation-v3

Conversation

@SunwardL

Copy link
Copy Markdown
Contributor

Summary

  • Add the evaluator-only P2-A v3 semantic Memory screening change.
  • Freeze the multilingual-e5-small-q8 identity, anonymous 44-record/80-query dataset, development grid, bootstrap gates, and holdout lock rules.
  • Keep production retrieval unchanged: no embedding model, vector index, or runtime dependency is added.

Changes

  • Add strict v3 dataset validation for scope/lifecycle labels, category quotas, zero-overlap positives, SHA-256 manifests, normalized duplicates, and cross-split near-duplicates.
  • Add evaluator-only candidates for semantic-gated RRF, lexical-veto reranking, lexical-veto with a preregistered margin, and terminology-map exploration.
  • Add shared authorization/lifecycle/K/prompt-budget input boundaries and fail-closed safety accounting for scope, lifecycle, authority, unknown, forbidden, fallback, and network violations.
  • Add deterministic three-repeat development-grid checks and development/holdout isolation tests.
  • Add local model fallback/resource evidence tests and update the v3 OpenSpec task record.

Evaluation and evidence

  • Base: upstream/develop@64b648e1 (PR chore(memory):archive-merged-retrieval-evaluation-changes #1314 merged).
  • Development run: 3 candidate modes × 54 frozen configurations, 40 development queries, zero network attempts, no holdout labels or metrics emitted.
  • Decision: no-go. The semantic-gated control fails abstention/safety; lexical-veto matches the lexical baseline without the preregistered gain; margin reranking regresses quality. No holdout lock is generated and P2-B remains blocked.
  • Refreshed evidence: D:\learning\Review_md\codex\p2a-semantic-spike\v3-development-output.json.
  • Dataset query hash: 8c56c3b665b4074a389d46f910caf4118a2b5620b91c8e8458c9afa456d5419f.

Verification

  • v3/related focused tests: 9 files, 32 tests passed.
  • build:kun, root npm run build, full npm run lint (0 errors), changed-file ESLint, file-lines, OpenSpec strict validation, and git diff --check passed.
  • Repository-wide npm run typecheck still reports the pre-existing missing phonemizer declaration in src/main/services/local-kokoro-worker-entry.ts; no production file was changed to mask it.

Scope and follow-up

  • This PR contains evaluation artifacts, tests, OpenSpec, and the development no-go record only.
  • Holdout evaluation is intentionally not run because no development candidate passed every preregistered gate.
  • Production embedding integration and P2-B remain blocked pending a separately approved candidate version.

@XingYu-Zhong
XingYu-Zhong merged commit 6a99c00 into KunAgent:develop Sep 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants