Skip to content

fix(ragas): score empty retrieval contexts as zero - #231

Open
Silu Panda (SiluPanda) wants to merge 1 commit into
braintrustdata:mainfrom
SiluPanda:fix/context-relevancy-empty-223
Open

Silu Panda (SiluPanda) wants to merge 1 commit into
braintrustdata:mainfrom
SiluPanda:fix/context-relevancy-empty-223

Conversation

@SiluPanda

Copy link
Copy Markdown
Contributor

Fixes #223.

ContextRelevancy now returns score 0 with an empty relevant-sentences list when the flattened context is empty. Python's sync and async paths and the TypeScript scorer short-circuit before calling the LLM judge. This prevents division by zero, NaN/null scores, or a hallucinated sentence turning empty retrieval into a perfect score.

The change leaves non-empty contexts and their existing score clamping unchanged. Tests cover "", [], and [""], assert that no judge request occurs, and check that the TypeScript result serializes as zero rather than null.

Validation:

  • Before the fix: 6 new Python cases and 3 new TypeScript cases fail because they call the judge for empty context.
  • Python: pytest py/autoevals/test_ragas.py -k 'not retrieval' -q gives 10 passed, 8 deselected.
  • TypeScript: the ContextRelevancy and mocked AnswerCorrectness tests give 6 passed, 3 skipped.
  • pnpm run build passes for CJS, ESM, and declarations.
  • All configured pre-commit checks on the four changed files pass.

Live-provider tests were deliberately not run; validation used local/mocked tests with API credentials unset.

AI assistance: implemented and validated using Codex.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant