Skip to content

Add LongMemEval retrieval benchmark#14

Merged
Da7-Tech merged 2 commits into
Da7-Tech:mainfrom
therealclvn:codex/longmemeval-benchmark
Jul 14, 2026
Merged

Add LongMemEval retrieval benchmark#14
Da7-Tech merged 2 commits into
Da7-Tech:mainfrom
therealclvn:codex/longmemeval-benchmark

Conversation

@therealclvn

@therealclvn therealclvn commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Closes #4.

Summary

  • add bench/longmemeval.py, a stdlib LongMemEval retrieval harness that maps official cleaned oracle histories to remember() and benchmark questions to recall()
  • measure exact turn-level evidence from the dataset's has_answer annotations, falling back to answer-session evidence only for sessions that have no marked answer turn
  • report evidence-turn@1/evidence-turn@k, answer-string@k, latency, dataset SHA-256, exact git commit, and the exact command used
  • download the official Hugging Face cleaned oracle JSON by default, while supporting --data for a local file or alternate URL
  • add a tiny LongMemEval fixture, regression coverage for evidence-label inflation, and a CI smoke command that avoids network downloads in the matrix

Maintainer audit correction

The initial harness counted every turn in an answer session as evidence. That made the public subset report evidence@1 = 1.000 even when recall returned an unrelated turn from the correct session. The audited revision now:

  • counts only marked has_answer turns at turn granularity
  • retains session-level fallback for the 94 official answer sessions that have no marked turn
  • excludes no-evidence instances from the reported average memory-record count
  • labels the metric explicitly as evidence-turn or evidence-session
  • adds regression tests for exact-turn evidence and unannotated-session fallback

Verification

  • python3 -m unittest discover -s tests — 272 passed
  • python3 bench/bench.py
  • python3 bench/multilang.py
  • python3 bench/discrim.py
  • python3 bench/soak.py
  • python3 bench/fuzz.py --quick
  • python3 bench/longmemeval.py --data tests/fixtures/longmemeval_tiny.json --limit 2
  • python3 bench/longmemeval.py --limit 50
  • python3 -m compileall mind.py bench/longmemeval.py tests/test_longmemeval_bench.py
  • git diff --check

Public LongMemEval subset result on the official cleaned oracle JSON:

dataset sha256: 821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c
commit: 979d3c7
command: python3 bench/longmemeval.py --limit 50
evaluated: 50 | excluded abstention: 30 | skipped no-evidence: 0
memory records: 978 total | 19.6 avg/question
evidence-turn@1 0.500 | evidence-turn@5 0.840 | answer-string@5 0.580

This is a retrieval benchmark, not LLM answer grading. The corrected lower evidence score is intentional: it reports retrieval of the actual annotated evidence turn instead of any turn from the right session.

@Da7-Tech

Copy link
Copy Markdown
Owner

Maintainer audit fix pushed in 979d3c7.

The original evidence@1 = 1.000 was inflated because every turn in an answer session counted as evidence. Against the same official 50-question subset, exact annotated-turn retrieval is 0.500 at rank 1 and 0.840 at rank 5. The harness now uses exact has_answer turns, with answer-session fallback only where the dataset has no turn annotation, and regression tests cover both paths.

All 272 tests and the full local benchmark/fuzz/soak set pass. Waiting for the refreshed cross-platform matrix before merge.

@Da7-Tech Da7-Tech merged commit c4b5259 into Da7-Tech:main Jul 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Independent benchmark: LoCoMo / LongMemEval subset

2 participants