Export runs as EvalPort ResultSets — portable output alongside outputs/*.json

[adhabnr-ux]

I read through src/memory_bench/models.py, judge.py, and dataset/base.py — nice, tight eval model. One gap I keep hitting when comparing AMB output to other harnesses' output: outputs/{dataset}/{memory}/{mode}/{split}.json is AMB-shaped only, so nothing outside AMB (or the external-results PRs like #24, #29, #32, #34) can consume a run without hand-parsing EvalSummary/QueryResult.

EvalPort is a small open interchange format for exactly this — a ResultSet JSON document any eval tool can read regardless of who produced it. AMB's model maps onto it almost 1:1:

AMB (models.py)	EvalPort (openeval.types)
Query.id / .query	TestCase.id / .input
Query.gold_answers	TestCase.expected_output (+ full list under metadata)
Query.gold_ids, .user_id, .meta	TestCase.metadata
GeminiJudge (judge.py)	one Grader (type="llm_judge")
QueryResult.query_id	Result.test_case_id
QueryResult.answer	Result.actual_output
QueryResult.correct, .score, .judge_reason	GraderResult.passed / .score / .reason
QueryResult.context_tokens, .retrieve_time_ms, .meta	Result.metadata
EvalSummary (dataset/split/memory_provider/mode/oracle/accuracy/…)	ResultSet.suite_id + .provider + .runner + .summary + .metadata

Sketch (would live in a new src/memory_bench/evalport.py, zero new hard deps — evalport-sdk has none):

from openeval.types import TestCase, Grader, Result, GraderResult, ResultSet, OPENEVAL_VERSION

def result_from_query_result(qr: QueryResult) -> Result:
    return Result(
        test_case_id=qr.query_id,
        passed=qr.correct,
        actual_output=qr.answer,
        grader_results=[GraderResult(
            grader_id="gr_amb_judge",
            type="llm_judge",
            score=qr.score if qr.score is not None else float(qr.correct),
            passed=qr.correct,
            reason=qr.judge_reason,
        )],
        metadata={"context_tokens": qr.context_tokens, "retrieve_time_ms": qr.retrieve_time_ms, **qr.meta},
    )

def resultset_from_summary(s: EvalSummary, run_id: str, started_at: str) -> ResultSet:
    return ResultSet(
        version=OPENEVAL_VERSION,
        suite_id=f"amb-{s.dataset}-{s.split}",
        run_id=run_id,
        started_at=started_at,
        results=[result_from_query_result(r) for r in s.results],
        provider={"model": s.answer_llm},
        runner={"name": "amb", "version": s.memory_provider},
        summary={"total": s.total_queries, "passed": s.correct,
                 "failed": s.total_queries - s.correct, "pass_rate": s.accuracy},
        metadata={"memory_provider": s.memory_provider, "mode": s.mode, "oracle": s.oracle,
                  "ingestion_time_ms": s.ingestion_time_ms, "judge_llm": s.judge_llm},
    )

Hooked up as amb run ... --export-evalport (writes a sibling .evalport.json next to the existing output), this would give every external-results contributor (#24 AutoMem, #29 Letta, #32/#36 ValorBrain, #33 MemoryHub) a standard artifact to attach that any EvalPort-aware tool can diff/replay, on top of the AMB-native JSON you already commit — no change to the existing format, purely additive.

For precedent on shipping this as a thin, standalone converter rather than a hard dependency: braintrust-openeval-adapter does the same to_openeval()/from_openeval() shape against Braintrust's Eval() result object — same idea, applied here to EvalSummary.

Happy to send a PR if this direction is useful (I'd start read-only: exporter only, no importer, no change to existing behavior).

— Sahi, independent contributor (not affiliated with this project)