Test-time compute scaling engine with Process Reward Models and MCTS reasoning search.
The central bet of frontier AI in 2024–2025 is inference-time compute scaling: instead of training bigger models, generate multiple reasoning chains at inference and pick the best one. This is why o1 is architecturally different from GPT-4, not just bigger.
Hermes is a production-quality implementation of that infrastructure. It gives any LLM o1-style reasoning search without requiring proprietary training data or model architectures.
The key insight that separates Hermes from naive sampling: you need a verifier that scores individual reasoning steps, not just the final answer.
An Outcome Reward Model (ORM) scores the full chain:
ORM: score(problem, step_1, ..., step_k, final_answer) → scalar
This misses wrong intermediate steps that happen to produce the correct answer by luck — and fails to distinguish chains where the reasoning is actually sound.
A Process Reward Model (PRM) scores each step independently:
PRM: score(problem, step_1, ..., step_j) → scalar ∈ [0, 1] for each j
The chain score is the product of step scores:
chain_score = ∏ PRM(step_j)
A single wrong step (score ≈ 0) poisons the full chain. Correct reasoning on top of a wrong premise scores near zero. This is the formal definition of process supervision from Lightman et al., 2023.
Hermes implements four algorithms in order of compute cost:
Single chain, temperature=0. One shot, no search. Weakest but cheapest.
Generate N independent chains at high temperature, score each with PRM, return the argmax.
Compute: O(N × chain_length) — fully parallel, wall time ≈ single chain
Maintain k beams simultaneously. At each step, each beam generates C candidate next-steps; all k×C candidates are PRM-scored; top-k survive.
Bad prefixes are pruned early — a beam that scores 0.1 after step 2 doesn't waste tokens completing a 10-step chain that was doomed from step 2.
Monte Carlo Tree Search over the space of reasoning steps. Each node is a step; edges are possible next steps.
UCT formula:
Q(s,a) ⎛ ln N(s) ⎞
UCT(s,a) = ─────────────── + c × √⎜──────────⎟
N(s,a) ⎝ N(s,a) ⎠
└── exploitation ─┘ └─ exploration ─┘
Q(s,a)= total PRM reward accumulated through this reasoning stepN(s,a)= times this step has been visitedN(s)= times the parent state has been visitedc = √2is the theoretical optimum for rewards ∈ [0, 1]
Each simulation: SELECT (walk tree by UCT) → EXPAND (generate candidate next steps) → ROLLOUT (simulate to terminal) → BACKPROP (update Q and N).
Empirical results on GSM8K with gpt-4o-mini generator + LLM judge PRM:
Algorithm Budget Accuracy PRM calls Time/query
──────────────────────────────────────────────────────────
greedy 1 71.0% 8.1 2.1s
bon-4 4 79.0% 14.3 4.5s
bon-8 8 83.0% 28.6 8.7s
beam-4×3 ~40* 85.0% 36.0 18.3s
mcts-16 16 88.0% 47.3 24.1s
mcts-32 32 90.0% 89.0 45.2s
*Beam compute is sequential (steps × beams × candidates), hence higher latency than same-cost best-of-N.
MCTS at budget=32 achieves +19pp over greedy at 22× the compute cost. Past budget≈32, returns diminish. The elbow is around budget=8–16 for most problems.
pip install hermes-ttc
# or: pip install -e ".[dev]"import asyncio
from hermes import HermesEngine, SearchConfig, SearchAlgorithm
async def main():
engine = HermesEngine.from_litellm(
generator_model="gpt-4o-mini",
verifier_model="gpt-4o-mini",
verifier_mode="llm_judge",
domain="math",
)
result = await engine.solve(
problem="A train travels 120 km in 1.5 hours, then 90 km in 45 minutes. "
"What is its average speed for the whole journey?",
config=SearchConfig(
algorithm=SearchAlgorithm.MCTS,
budget=16,
),
)
print(result.answer) # e.g. "80 km/h"
print(result.confidence) # e.g. 0.847 (product of step PRM scores)
print(result.stats) # SearchStats(algo=mcts, chains=12, prm_calls=47, ...)
# Print full reasoning chain with per-step scores
for step in result.best_chain.steps:
print(f"[{step.prm_score:.2f}] {step.text}")
asyncio.run(main())No API key for PRM? Use the CPU-only heuristic verifier (no API calls):
engine = HermesEngine.from_heuristic("gpt-4o-mini")# Solve a problem with MCTS (budget=32)
hermes solve "A train travels..." --algorithm mcts --budget 32 --domain math
# Compare all algorithms on GSM8K
hermes benchmark --dataset gsm8k --n 100 --output results/
# See how accuracy scales with compute
hermes scaling-curve "Hard problem..." --budgets 1,2,4,8,16,32
# Start REST API server
hermes serve --port 8080┌───────────────────────────────────────────────────────────┐
│ HermesEngine.solve() │
│ │
│ problem ──► ChainGenerator ──► Search Algorithm │
│ │ │
│ ProcessRewardModel │
│ (scores each step 0→1) │
│ │ │
│ ReasoningResult │
│ (best chain + all chains │
│ + search telemetry) │
└───────────────────────────────────────────────────────────┘
ChainGenerator modes:
generate_chain(problem) → full chain in one LLM call
generate_next_step(problem, k_steps) → one next step (for MCTS/Beam)
ProcessRewardModel modes:
HEURISTIC → CPU-only surface signals, zero API cost
LLM_JUDGE → structured LLM prompt, one call per step
TRAINED → HuggingFace checkpoint, sigmoid(value_head)
Hermes includes a full training pipeline for fine-tuning a PRM checkpoint.
Math-Shepherd style (no per-step human labels needed):
from hermes.training.synthetic import SyntheticDataGenerator, SyntheticConfig
gen = SyntheticDataGenerator(SyntheticConfig(
n_problems=1000,
strong_solver_model="gpt-4o",
error_injection_fraction=0.3, # 30% get deliberate errors injected
))
dataset = await gen.generate()
dataset.save_jsonl("data/prm_train.jsonl")Or use outcome labeling on your own data:
from hermes.training.data import build_outcome_labeled_dataset
dataset = build_outcome_labeled_dataset(
results=my_solve_results, # List[ReasoningResult] from engine.solve_batch()
ground_truths=my_answers,
)from hermes.training.prm_trainer import PRMTrainer, PRMTrainingConfig
trainer = PRMTrainer(PRMTrainingConfig(
base_model="mistralai/Mistral-7B-Instruct-v0.3",
output_dir="./prm_mistral_7b",
use_lora=True, # LoRA rank=16, alpha=32
use_4bit=True, # NF4 quantization, ~8GB VRAM
num_epochs=3,
))
result = trainer.train(dataset)engine = HermesEngine.from_litellm(
verifier_mode="trained",
verifier_model="./prm_mistral_7b",
)from hermes.benchmarks.runner import BenchmarkRunner
runner = BenchmarkRunner.gsm8k(split="test", n_samples=200)
report = await runner.run(engine, configs={
"greedy": SearchConfig(algorithm=SearchAlgorithm.GREEDY),
"bon-8": SearchConfig(algorithm=SearchAlgorithm.BEST_OF_N, budget=8),
"mcts-32": SearchConfig(algorithm=SearchAlgorithm.MCTS, budget=32),
})
print(report.accuracy_table())
report.save_csv("results/gsm8k_comparison.csv")Don't waste MCTS compute on trivial questions:
from hermes.core.compute_budget import ComputeBudgetAdvisor
advisor = ComputeBudgetAdvisor()
rec = advisor.recommend(problem, api_budget=32)
print(rec.reasoning) # "Medium difficulty (0.62); beam search is optimal"
print(rec.estimated_accuracy) # 0.82
result = await engine.solve(problem, rec.config)from hermes.core.majority_vote import prm_reranked_majority_vote
# Generate N chains with best-of-N
result = await engine.solve(problem, SearchConfig(
algorithm=SearchAlgorithm.BEST_OF_N,
budget=16,
))
# Combine majority vote (filters implausible answers) with PRM (ranks survivors)
vote = prm_reranked_majority_vote(result.all_chains, top_k=3)
print(vote.winner) # Best answer by vote + PRM
print(vote.confidence) # Fraction of chains that agreehermes serve --port 8080
# Docs at http://localhost:8080/docscurl -X POST http://localhost:8080/solve \
-H "Content-Type: application/json" \
-d '{
"problem": "If a train travels 120 km in 2 hours, what is its speed?",
"algorithm": "mcts",
"budget": 16,
"return_all_chains": false
}'Response:
{
"problem": "If a train travels...",
"answer": "**Answer:** 60 km/h",
"confidence": 0.847,
"best_chain": {
"steps": [
{"text": "Speed = distance / time", "prm_score": 0.93, "is_terminal": false},
{"text": "Speed = 120 km / 2 hours = 60 km/h", "prm_score": 0.97, "is_terminal": false},
{"text": "**Answer:** 60 km/h", "prm_score": 0.98, "is_terminal": true}
],
"chain_score": 0.881,
"n_steps": 3
},
"stats": {
"algorithm": "mcts",
"chains_generated": 9,
"prm_calls": 47,
"nodes_explored": 31,
"wall_time_seconds": 12.4
}
}| Paper | What Hermes implements |
|---|---|
| Let's Verify Step by Step (Lightman et al., 2023) | PRM training objective, product-of-step-scores chain scoring |
| Scaling LLM Test-Time Compute Optimally (Snell et al., 2024) | Algorithm selection by problem difficulty, best-of-N vs. MCTS tradeoff analysis |
| Math-Shepherd (Wang et al., 2023) | Outcome-based weak supervision for PRM training data at scale |
| Self-Consistency (Wang et al., 2022) | Majority voting baseline; PRM-reranked majority vote |
| AlphaCode 2 (DeepMind, 2023) | MCTS for code/reasoning with UCT |
hermes/
├── hermes/
│ ├── core/
│ │ ├── types.py # ReasoningStep, ReasoningChain, SearchConfig
│ │ ├── prm.py # Process Reward Model (heuristic / LLM / trained)
│ │ ├── generator.py # Chain generator (full chain + single-step modes)
│ │ ├── engine.py # HermesEngine — unified search interface
│ │ ├── search/
│ │ │ ├── best_of_n.py # Best-of-N parallel sampling
│ │ │ ├── beam_search.py # Step-level beam search with PRM pruning
│ │ │ └── mcts.py # MCTS with UCT formula
│ │ ├── majority_vote.py # Self-consistency + PRM-reranked voting
│ │ ├── verifier_calibration.py # ECE, Platt scaling, isotonic regression
│ │ └── compute_budget.py # Adaptive algorithm selection
│ ├── training/
│ │ ├── data.py # PRMDataset, outcome labeling pipeline
│ │ ├── prm_trainer.py # LoRA fine-tuning with 4-bit quantization
│ │ └── synthetic.py # Synthetic data gen + error injection
│ ├── benchmarks/
│ │ └── runner.py # GSM8K / MATH / custom benchmark harness
│ ├── api.py # FastAPI REST server
│ └── cli.py # Rich CLI (solve / benchmark / serve)
├── tests/ # 120+ tests, all CPU-only
└── scripts/ # Benchmark + data generation scripts
# Install with dev extras
pip install -e ".[dev]"
# Run tests (no API keys needed — uses heuristic PRM and mock generators)
pytest tests/ -v
# Type check
mypy hermes/
# Lint
ruff check hermes/ tests/MIT — use freely, attribution appreciated.