evaluator-optimizer-loop.mdx

title	Evaluator-Optimizer loop
tagline	Generator writes code; an automated evaluator (tests, lint, types) returns specific failures; loop until green.
attribution	Anthropic (Building Effective Agents)
tier	snippet
canonical_url	https://www.anthropic.com/research/building-effective-agents
upstream	repo ref paths anthropics/anthropic-cookbook main
when_to_use	Anything with a machine-checkable pass/fail signal, build until green, lint until clean, perf budget met.
when_not_to_use	Tasks without a deterministic check (open-ended prose, design decisions).
tags	primitive anthropic loop tests-as-evaluator
inputs	A task with a clear quality signal (test pass/fail, lint, type-check, reviewer-rubric) An evaluator prompt An optimizer/agent prompt
review_gate	trust standards merge description tool-assisted tool-assisted human-gate Evaluator approves the optimizer output before merge; human approves the final PR.
checkpoints	phase description per-iteration Evaluator runs after every optimizer attempt
use_cases	building-features test-generation bug-fixing security-remediation
sources	title author url year Building Effective Agents Anthropic https://www.anthropic.com/research/building-effective-agents 2024 title author url year Evaluator-Optimizer (Anthropic Cookbook) Anthropic https://github.com/anthropics/anthropic-cookbook/blob/main/patterns/agents/evaluator_optimizer.ipynb 2024
related	patterns antiPatterns practices glossary workflows evaluator-optimizer reflexion-self-critique-loop llm-as-judge-eval self-validating-tests test-overfitting-test-cheating verify-everything graders-resistant-to-hacking agentic-loop reflexion-loop tdd-red-green-refactor-skills ralph-wiggum-loop
loop	trigger steps gate exit back Task Generator writes code Evaluator runs tests, lint, types All green? Ship failures

The simplest workflow primitive that actually closes a loop. An optimizer writes code against a spec; an evaluator runs an automated check (tests, type-checker, lint, perf budget) and returns specific failures; the optimizer revises. Repeat until green or until the iteration budget runs out.

while iteration < MAX:
    code  = optimizer.run(spec, prev_feedback)
    score = evaluator.run(code)        # pytest / tsc / ruff / bench
    if score.passed: break
    prev_feedback = score.failures

The technique's strength is that the evaluator is deterministic and machine-checkable. No LLM judges code quality; tests do.

Use for anything with a real pass/fail signal: build-until-green TDD, lint-until-clean refactors, "make this benchmark beat 100ms." A tight feedback loop with concrete failure messages is the cleanest agent contract in this catalog.

Avoid for open-ended generation where there's no machine-checkable correctness signal. Use Reflexion-style critic loops there instead.

Pitfall: the evaluator can be reward-hacked, an agent will sometimes "fix" a failing test by editing the test rather than the code. Pin tests as immutable in the optimizer's permissions, or run a final review pass before merge.

Claude Code one-liner, runnable today:

claude -p "Run pytest. If anything fails, fix only the source code (never the tests). Repeat until all tests pass. Do not edit any file in tests/."