Skip to content

README.md

Latest commit

 

History

History
79 lines (62 loc) · 3.27 KB

README.md

File metadata and controls

79 lines (62 loc) · 3.27 KB
Raw

Experience to Competence Without Weight Updates

DOI

Pre-registered falsifiable gates for agent memory — everything runs locally on a laptop (M1 Pro, Ollama). Paper: doi.org/10.5281/zenodo.21323673 (paper/main.tex in this repo).

TL;DR

We built a biologically inspired memory system (episodic traces, repetition strengthening, consolidation) and tried to kill it five times with pass/kill criteria committed before each run. Everything is reported, including the kills:

Gate Question Verdict
M0 Does retrieval help at all? PASS — naked 0.00 → memory 0.98
M1 Does repetition strengthen recall? curve real, but see M3
M2 Does consolidation compress without forgetting? PASS — 350 exposures → 150 nodes, retention unchanged
M3 Does the "brain" beat vanilla RAG? KILL ×2 — strength ranking has negative net value; removed from default
M4 Does experience become competence, frozen weights? PASS — first-try 0.45 → 1.00 → 1.00, control flat 0.00, oracle matched

Two measured side-results:

  • Cosine similarity cannot deduplicate paraphrases: distributions of same-fact paraphrases and same-template distractors fully overlap (medians 0.83 vs 0.87). Any embedding-only dedup is structurally unsound; we use a local LLM same-fact judge instead.
  • A mechanism can produce a beautiful learning curve while hurting: M1's repetition curve looked like a discovery until M3's control arm showed the interference it "rescued" facts from was self-inflicted.

The M4 result

A frozen 9B agent (qwen3.5, no gradient ever) must satisfy a hidden-rule JSON validator (13 discoverable rules, first-error-only feedback, fresh data each episode). Lessons = verbatim error messages stored in the memory, recalled each attempt.

Arm Block 1 Block 2 Block 3
Control (no memory) 0.00 0.00 0.00
Memory 0.45 1.00 1.00
Oracle (rules given) 1.00 0.90 0.95

60 episodes, n=20/block, bootstrap CIs in results/gate_m4.json. Seed replication: transfer vs control categorical in 3/3 seeds; block-curve significance in 1/3 (details in the paper).

Reproduce

# prerequisites: Ollama with llama3.1, qwen3.5:9b, nomic-embed-text
uv venv && uv pip install -e .
export SB_VAULT_PATH=/tmp/sb_vault SB_DB_PATH=/tmp/sb.sqlite3
python -m pytest tests/          # 34 tests
python scripts/gate_m4.py --episodes 60 --seed 42

Every gate script carries its pre-registered criteria in the module docstring. Per-run JSON results and honest write-ups (including failed runs) are in results/.

Cite

@misc{raviotta2026experience,
  title  = {Experience to Competence Without Weight Updates:
            Pre-Registered Falsifiable Gates for Agent Memory},
  author = {Raviotta, Matthias},
  year   = {2026},
  doi    = {10.5281/zenodo.21323673},
  url    = {https://doi.org/10.5281/zenodo.21323673}
}

Tri-licensed (RSALv2 / SSPLv1 / AGPLv3), like Atlas — see LICENSE.txt. Built in the open by the maker of Atlas · atlasquant.app