Add support for classifier "scorers"

[aswink]

No description provided.

[aswink]

• Add support for classifier "scorers" #1553 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Abhijeet Prasad (AbhiPrasad)]

This also needs a bunch more tests, but we can tackle that after this.

[Abhijeet Prasad (AbhiPrasad)]

should we make this purpose: "classifier" for classifier spans? We might need to introduce a new runInClassiferSpan helper.

[aswink]

I'm not totally sure of the implications of this, I think it would require a bunch of backend changes as well if we did. Could be a good idea, though, Alex Z (@CLowbrow) do you know?

[Abhijeet Prasad (AbhiPrasad)]

More context from Ankur - this is a PITA to change because there is a bunch of places in the backend that rely on it. We can't do it. Relevant slack thread: https://braintrustdata.slack.com/archives/C083GCUTVDZ/p1773705541982479

[Abhijeet Prasad (AbhiPrasad)]

warning AI slop:

I ran this PR against my spec in https://github.com/braintrustdata/braintrust-spec/pull/2 (after cleaning up stuff like the confidence changes) and it codex gave me the two following things:

• The TypeScript API contract is weaker than the spec. The spec models Evaluator as a union that requires either scores or classifiers at the type level (spec lines 23-31:23). The PR makes both optional in js/src/framework.ts and only enforces the rule at runtime. It also keeps scores required on EvalResult, so classifier-only runs likely still produce scores: {} even though the spec makes scores optional.
• The SDK toolchain is not classifier-aware end to end. The execution path in js/src/framework.ts supports classifiers, but js/src/cli/functions/upload.ts, js/dev/server.ts, and js/src/cli/functions/infer-source.ts were only updated to tolerate missing scores; they still enumerate, bundle, and resolve only scorers. So classifier support is partial: local eval execution works, but upload/dev/source-inference surfaces are not fully wired up.

These seem like things we have to tackle.