Skip to content

fix(evaluation): support bilingual DE-Arch scores - #10

Open
linhongyu510 wants to merge 2 commits into
ByteDance-Seed:mainfrom
linhongyu510:fix/de-arch-bilingual-scoring
Open

fix(evaluation): support bilingual DE-Arch scores#10
linhongyu510 wants to merge 2 commits into
ByteDance-Seed:mainfrom
linhongyu510:fix/de-arch-bilingual-scoring

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Aug 14, 2026

Copy link
Copy Markdown

Summary

Fix bilingual scoring in the DE-Arch evaluation suite and preserve accurate score denominators when an input cannot be judged.

The shipped English and Chinese gold files use different score labels, and eight English rubrics are truncated (002, 006, 007, 012, 016, 017, 019, and 024). This could make English tasks score zero or be judged against incomplete criteria.

This change:

  • parses both Total Score / Points and 总分 / 分数 fields
  • supports suffixed Chinese task names, project-level language hints, and legacy unsuffixed English tasks
  • uses the complete paired Chinese gold only when the matching English rubric is incomplete
  • rejects a task when no complete rubric is available
  • scores empty blueprints as zero without calling the LLM judge
  • keeps known rubric maxima in the denominator for failed or empty tasks
  • resolves default gold and result paths relative to the evaluation suite instead of the process working directory

Validation

  • pytest tests/test_de_arch_evaluation.py -q — 15 passed
  • Python compilation check
  • git diff --check

The tests cover bilingual score extraction, language selection, truncated-rubric fallback, missing-rubric rejection, empty blueprints, denominator preservation, and all 30 shipped DE-Arch task pairs.

AI assistance

This change was developed with AI assistance. I reviewed the shipped gold data, verified the incomplete rubric set, and ran the checks above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant