ContextRelevancy divides by context length with no zero guard: an empty context gives NaN (serialized as the "skipped" null) or a perfect 1 in JS, and ZeroDivisionError in Python #223

js/ragas.ts:280-293 (inside export const ContextRelevancy, which starts at line 247):

const sentences = relevantSentencesSchema.parse(mustParseArgs(response));
// Clamp score to [0, 1] - the LLM may return sentences longer than the
// original context due to paraphrasing or hallucination (#80)
const rawScore =
  sentences.sentences.map((s) => s.sentence).join("").length /
  context.length;
return {
  name: "ContextRelevancy",
  score: Math.min(Math.max(rawScore, 0), 1),
  metadata: { relevantSentences: sentences.sentences },
};

To reach this code, context only has to be defined. checkRequired (js/ragas.ts:989-1001) rejects only undefined, so context: "" passes. So does context: [], which flatenContext (js/ragas.ts:981-987) joins to "". A retriever that returns zero documents produces exactly this input.

Two things can happen next:

• The judge extracts nothing. The prompt tells it to do this when nothing is relevant: "If no relevant sentences are found ... return an empty array" (js/ragas.ts:224). Then rawScore is 0 / 0. Math.max and Math.min do not clear NaN, so the function returns { score: NaN }. It does not throw and sets no error field.
• The judge returns any text at all. This is the paraphrase or hallucination behaviour behind Context Relevancy issue with score not between 0 and 1. #80. Then rawScore is x / 0 = Infinity, which the clamp turns into 1. An empty retrieval gets a perfect relevancy score.

The Python port, py/autoevals/ragas.py:397-409, uses the same formula with no guard:

def _postprocess(self, context, response):
    sentences = json.loads(response["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"])
    raw_score = len("".join([s["sentence"] for s in sentences["sentences"]])) / len(context)
    return Score(
        name=self._name(),
        score=min(max(raw_score, 0), 1),
        metadata={"relevant_sentences": sentences["sentences"]},
    )

Its check_required (py/autoevals/ragas.py:127-130) checks only is None. Both _run_eval_sync and _run_eval_async join a list context with "\n", so [] becomes "" here too.

Measured

To reproduce, the scoring formula and the context-flattening step were copied verbatim into fresh Node and Python processes. No repository module was imported and there was no network access.

The irrelevant-context control shows that the bad values come from context.length being 0, not from an empty sentence list.

Consequence

py/autoevals/score.py:16-17 documents the contract: "If the score is None, the evaluation is considered to be skipped." The JS Score type mirrors it as score: number | null (js/score.ts:5).

JSON.stringify(NaN) is null. When ContextRelevancy gets an empty context and extracts nothing, its serialized output is identical to a scorer that chose to abstain. A consumer that follows the documented contract drops that row from the mean instead of counting it as a failure. An empty retrieval should score 0 for relevancy. When that row is dropped, and the mean of the other rows is above 0, the reported average goes up.

In the other empty-context case, the JS port reports a perfect 1 for a retrieval that returned nothing.

In Python the same input raises a bare ZeroDivisionError from inside _postprocess. Errors are documented to propagate (py/autoevals/score.py:22-24), so this at least fails loudly. The message, however, does not name the scorer or the empty context. The two ports also behave differently on identical input.

The mocked unit tests for ContextRelevancy in js/ragas.test.ts and py/autoevals/test_ragas.py cover only the clamp above 1 (#80) and one normal-case fixture. The live-API parametrized tests use a non-empty context. No test uses an empty context.

Suggested fix

The sibling scorer in the same file, ContextRecall._postprocess (py/autoevals/ragas.py:567), already guards this kind of division: score=ones / total if total > 0 else 0.

ContextRelevancy in both ports should check context.length === 0 / len(context) == 0 before dividing. It should then either:

• return a defined 0, since an empty context cannot be relevant to anything, or
• raise a clear, named error, in the style of checkRequired: "ContextRelevancy requires a non-empty context".

Either choice rules out NaN, the null that looks like a skip, and the clamped Infinity that becomes 1. A test with context: "" and context: [] in each port would pin the result.

Related: open PR #210 ("fix(json): stop a skipped sub-score being averaged as a mismatch") deals with the same kind of problem in JSONDiff: a computed result that is indistinguishable from an intentional skip.

Happy to open the PR.