Skip to content

refactor(compression): replace compact prompt with structured execution-state format - #2284

Open
ArrogHie wants to merge 1 commit into
GCWing:evals-on-releasefrom
ArrogHie:evals-on-release
Open

refactor(compression): replace compact prompt with structured execution-state format#2284
ArrogHie wants to merge 1 commit into
GCWing:evals-on-releasefrom
ArrogHie:evals-on-release

Conversation

@ArrogHie

Copy link
Copy Markdown
Contributor

Summary

Replace the compaction prompt in �uild_compact_prompt (src/crates/assembly/core/src/agentic/session/compression/compressor.rs) with a compression-oriented prompt that produces a durable, structured execution state.

Changes

  • The new prompt instructs the model to compress all available context into one block containing five sections: OBJECTIVE, REQUIREMENTS, EXECUTION STATE, CONTINUATION, and CRITICAL EXACT DETAILS.
  • Replaces the previous conversation-recap style prompt (which required a transient block plus a 9-section recap) with hard rules, a compression procedure, recursive-compression safety rules, requirement handling with stable IDs, and a final verification checklist.
  • The compaction-contract injection (authoritative factual context from tool observations) is preserved at the end of the prompt.

Verification

  • No other code or tests in the workspace reference the removed prompt wording.
  • Existing test model_summary_prompt_includes_compaction_contract remains compatible (contract text is still injected).

…on-state format

Replace the conversation-recap style compaction prompt in
build_compact_prompt with a compression-oriented prompt that
produces a durable, structured execution state (OBJECTIVE /
REQUIREMENTS / EXECUTION STATE / CONTINUATION / CRITICAL EXACT
DETAILS) inside a single <summary> block. The compaction-contract
injection is preserved.
@ArrogHie

ArrogHie commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

glm-5.2 完整上下文压缩提示词评测报告

模型:glm-5.2(openbitfun 网关,流式传输 + thinking budget=10000)
评分方式:结构化直评(structured_v1,确定性解析 candidate_summary 分段结构后与 expected_state 结构化对比)
截断比例:50% / 75%(对整条轨迹截取,前缀完整未截断)
测试集:10 个真实 BitFun Agent 长程任务成功轨迹 case x 2 比例 = 20 组合/提示词
评测对象:baseline / v3 / v4 / v5 共 4 个压缩提示词
生成时间:2026-08-03

1. 理解度评分汇总

UnderstandingScore = 0.25目标理解 + 0.25进展理解 + 0.20约束保真 + 0.15失败方案召回 + 0.15*关键事实召回

提示词 r=0.5 均值 r=0.75 均值 总均值 摘要均长
prompt-baseline 0.432 0.357 0.395 13837
prompt-v3 0.405 0.278 0.341 14349
prompt-v4 0.451 0.395 0.423 11921
prompt-v5 0.497 0.404 0.450 9675

3. 五维度分解

维度 baseline v3 v4 v5
目标理解度 0.530 0.440 0.570 0.568
进展理解度 0.256 0.259 0.257 0.355
约束保真度 0.422 0.259 0.433 0.539
失败方案召回 0.637 0.642 0.678 0.600
关键事实召回 0.120 0.121 0.184 0.144

5. 压缩效率

摘要长度 vs 前缀长度,反映压缩激进程度。

提示词 r=0.5 摘要/前缀 r=0.75 摘要/前缀 平均压缩率
prompt-baseline 5.9% 6.3% 6.1%
prompt-v3 6.4% 6.0% 6.2%
prompt-v4 5.6% 4.8% 5.2%
prompt-v5 4.3% 4.0% 4.1%

6. 关键发现

  1. 总体理解度:baseline=0.395,v3=0.341,v4=0.423,v5=0.450。
    v5 以 0.450 领先所有版本(+0.056 vs baseline),v4 次之(0.423,+0.028),v3 下降至 0.341(-0.053)。
  2. 压缩效率:baseline 摘要均长 13837 字符,v3 14349,v4 11921,v5 9675。
  3. 稳定性(标准差):baseline=0.137,v3=0.153,v4=0.143,v5=0.147。
  4. v5 以更少的摘要保留了更多的重要信息

exp25(300K 窗口全量+v5提示词优化) vs baseline(1M 窗口全量)对比报告

  • 日期:2026-08-14

1. 实验配置

配置项 exp25(本次) exp12(基线)
模型 deepseek-v4-flash deepseek-v4-flash
思考强度 max max
上下文窗口 300K 1M(1048576)
CLI bitfun-cli-pv5-aotuCmp(85747881) bitfun-0.2.14(de9eb3721)
轮数上限 --max-rounds 0(无上限) 默认 200
任务集 全量 113 全量 113

2. 总体指标对比(113 任务,exp25 全部有效)

指标 exp25 (300K) exp12 (1M) 变化
成功率(reward=1) 54/113 = 47.8% 51/113 = 45.1% +2.7pp
f2p 均值 0.8807 0.8812 -0.0005(持平)
p2p 均值 0.9975 0.9999 -0.002
partial 均值 0.9571 0.9572 -0.0001
input token 总量 2193.0M 2578.2M -14.9%
cache token 总量 2127.1M 2563.6M -17.0%
output token 总量 15.4M 14.3M +7.7%
cache 命中率 0.9699 0.9943 -0.024
成本(公式计价) $139.31 $94.47 +47.5%
压缩次数 52(52 个任务) 0(旧 CLI 无压缩)
峰值上下文均值 204.2K 230.0K -11%
agent 步数均值 141.9 130.1 +9%
耗时合计(per-trial 求和) 81.3h 54.5h +49%
单任务耗时均值 43.2 min(本机运行性能差距大) 28.9 min +49%

成本公式:cost = cache×$0.02/M + (input−cache)×$1/M + output×$2/M

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant