fix(matchers): strip leading headers in context relevance grading - #10568
fix(matchers): strip leading headers in context relevance grading#10568AbhiPra24 wants to merge 1 commit into
Conversation
Strip leading markdown headers and preamble labels before splitting sentences to prevent headers from inflating the sentence count denominator or score. Fixes promptfoo#10245
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33700b2cc3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const hasSubsequentContent = lines.slice(startIdx + 1).some((l) => l.trim().length > 0); | ||
| if (hasSubsequentContent) { | ||
| startIdx++; | ||
| continue; |
There was a problem hiding this comment.
When context is a string[], matchesContextRelevance counts every non-empty chunk unchanged, but the grader output still passes through this branch. If the first relevant chunk is # Refund policy and the grader echoes it followed by another relevant chunk, this drops the first chunk merely because later output exists, producing a score of 1/2 instead of 2/2. Restrict stripping to recognized grader preambles or normalize the numerator and denominator identically.
AGENTS.md reference: src/matchers/AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
| const PREAMBLE_LABEL_PATTERN = | ||
| /^(?:#{1,6}\s+)?(?:\*{1,2}|_{1,2}|`+)?\s*(?:(?:candidate|extracted|relevant|selected)\s+(?:(?:sentences?|context|passages?|information|units?)\b(?:\([sS]\))?)|context\s+relevance\b)(?:\s*[:])?(?:\*{1,2}|_{1,2}|`+)?\s*:?\s*/i; |
There was a problem hiding this comment.
Because the colon is optional and the pattern is not restricted to a label-only line, normal sentences such as Relevant information about Paris is here. or Context relevance matters here. are rewritten to about Paris is here. and matters here.. This splitter is also used for context-faithfulness statements, so those damaged statements are sent to the NLI grader and can change faithfulness verdicts; allow delimiter-free matching only when the whole line is the label.
AGENTS.md reference: src/matchers/AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
Reviewed changes to context relevance matching and shared text utilities. The PR adds header/preamble stripping before sentence splitting and uses the cleaned context in segmentation. No new LLM calls, tools, or capabilities were introduced, and output handling remains non-executable. No LLM security vulnerabilities were identified in these changes.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #10568 +/- ##
=======================================
Coverage 82.54% 82.54%
=======================================
Files 944 944
Lines 80640 80687 +47
Branches 26959 26974 +15
=======================================
+ Hits 66562 66605 +43
- Misses 14078 14082 +4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
1 participant
Description
Strip leading markdown headers and preamble labels (e.g.,
# Extracted sentences,## Relevant Context:,**Candidate sentences:**,Candidate sentences:, etc.) before splitting sentences so that grader response headers or context headers do not artificially inflate the sentence count denominator or score.Fixes #10245
Testing
test/matchers/shared.test.tsforstripLeadingHeaders,splitIntoSentences, andsplitTextIntoSentences.test/matchers/context-relevance.test.tsverifying context relevance scores without header pollution.