Idea: EvalPort converter for `frontier harbor trial --json` / research evaluator.py output

[adhabnr-ux]

Hi — I maintain EvalPort, a small open interchange schema for LLM eval data (TestCase/Grader/Result/ResultSet/GraderResult, with Python and TypeScript SDKs). Came across Frontier-CS through the Harbor blog post and read through the algorithmic/research tracks and the adapters/ Harbor adapters — really like that both tracks already emit structured per-run JSON rather than just a pass/fail line.

Wanted to float a small, concrete idea rather than a vague "you should support X":

What I saw:

• Research track: evaluator.py (e.g. research/problems/flash_attn/evaluator.py) writes a result.json with status, score, score_unbounded, pass_all, plus problem-specific fields like geometric_mean_speedup, total_tests/passed_tests.
• Harbor track: frontier harbor trial algorithmic 0 -a codex -m gpt-5.5 --json (per the README) returns reward, score, score_unbounded, trial_status, n_input_tokens/cost_usd, successful_submissions.
• Algorithmic track: partial credit comes from a chk.cc testlib.h checker (quitp(score_ratio, "Ratio: %.4f ...")) against config.yaml subtasks — not a fixed-answer grader, but a custom scorer.

None of that maps to "exact_match against expected_output" — which is exactly the case EvalPort's Grader.type: "custom" + params.handler exists for, and why GraderResult/Result carry a free-form metadata dict rather than assuming a rubric shape.

Sketch of the mapping (Python SDK is sdk/python/openeval, published as evalport-sdk on PyPI):

from openeval.types import Result, GraderResult, ResultSet, OPENEVAL_VERSION

def harbor_trial_to_result(trial: dict) -> Result:
    # trial is one `frontier harbor trial ... --json` object
    return Result(
        test_case_id=trial["task_name"],           # e.g. "frontier-cs/frontier-cs-algorithm-0"
        passed=trial["score"] > 0,
        grader_results=[GraderResult(
            grader_id="gr_frontier_cs_checker",
            type="custom",
            score=trial["score"] / 100.0,           # FrontierCS is 0-100; EvalPort scores are 0-1
            passed=trial["score"] > 0,
            metadata={
                "score_unbounded": trial.get("score_unbounded"),
                "reward": trial.get("reward"),
                "cost_usd": trial.get("cost_usd"),
                "successful_submissions": trial.get("successful_submissions"),
            },
        )],
        error={"message": trial["error_message"]} if trial.get("error_message") else None,
        metadata={"agent": trial.get("agent"), "model": trial.get("model")},
    )

def to_resultset(trials: list[dict], run_id: str) -> ResultSet:
    return ResultSet(
        version=OPENEVAL_VERSION, suite_id="frontier-cs/algorithmic",
        run_id=run_id, started_at=..., results=[harbor_trial_to_result(t) for t in trials],
    )

Same shape works for the research track's result.json (one Result per problem instead of per Harbor trial).

The practical payoff: weekly-eval.yml runs already produce these JSON blobs per problem/model — converting them to ResultSets would let anyone compare a Frontier-CS run against results from a different harness side-by-side (same Result/GraderResult shape), without changing anything about how Frontier-CS scores solutions.

There's precedent for exactly this "read the tool's native JSON, don't touch the tool" pattern in EvalPort's adapters/ — e.g. braintrust-openeval-adapter does the same normalize-then-wrap for Braintrust's Eval() result objects.

Happy to put together a frontier-cs-openeval-adapter (mirroring the adapters/frontier-cs-algorithm / adapters/frontier-cs-2.0 layout already in this repo) as a PR if that's of any use — or if this isn't a direction you want to take Frontier-CS's output format in, no worries at all, feel free to close.

— Sahi, independent contributor (not affiliated with this project)