(MOT-4680) fix(workers): describe every function the way agents ask for it, and drop the reranker floor - #1096
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedToo many files! This PR contains 176 files, which is 76 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (176)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 73 skipped (no docs/).
Four for four. Nicely done. |
andersonleal
force-pushed
the
feat/minilm-followups
branch
from
September 4, 2026 19:13
cb784a1 to
b44239f
Compare
…idate is noise
The gap cut trims padding under a real leader but never questioned the
leader itself, so a query nothing serves ("today's top news stories") still
filled its page with dense lookalikes. PRODUCTION_RERANK_FLOOR = -9 logits:
a head whose best cross-encoder score is under it keeps the BM25 ranking
(empty for most no-match wording). Measured on the 79-case set: match
51→49, multi 7→6, no-match false positives 4→3, the news case 11→2
functions. The two lost matches are hard paraphrases whose target scores
like noise; no logit separates them from an unserved query.
Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…ig-change handler internal harness (a dev-dependency) moved to iii-sdk 0.23.0 without approval-gate's Cargo.lock following, so every `--locked` cargo invocation on this crate failed before compiling. `cargo update -w` adds the second iii-sdk copy the graph already needed; the crate's own pin stays at 0.22.1-alpha.25. `approval::on-config-change` now carries `metadata.internal: true`, the convention directory::search_functions and the registry card use to keep plumbing out of agent-facing results. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…d Windows MSVC Cargo.toml's target cfg decides whether ort/fastembed are dependencies and build.rs' MINILM_TARGETS decides whether the dense lane compiles; a target on which they drift builds green as BM25-only, and only x86_64 linux-gnu had a CI build at all. A matrix job on macos-latest and windows-latest links the static ONNX Runtime and runs tests/minilm_target with III_REQUIRE_MINILM set, which fails unless cfg(minilm) is on. The console UI is stubbed so build.rs skips pnpm; it plays no part in the proof. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
… depth 24 Measured on the 79-case qrels set with the PR #1084 benchmark harness (catalog snapshot 20260902T165245039380Z), floor disabled so the knobs show their own effect: reranker weight 1.25 -> 1.0 top-1 42 -> 44, MRR@1 0.778 -> 0.815, paraphrase MRR@1 0.583 -> 0.667; 2.0 and 4.0 go the other way (40 top-1 at 4.0) gap 10 -> 12 match 51 -> 52/54 (recovers the undo paraphrase), recall@12 0.910 -> 0.938, 8.7 functions per query instead of 7.7 depth 48 -> 24 rankings identical to 48 and 96; half the cross-encoder pairs (benchmark wall time 45.6 s -> 32.7 s) RRF K 10 / 20 / 30 / 60 no effect on any metric; left at 60 Combined, floor off: match 52/54, top-1 44, recall@12 0.938, nDCG@12 0.860. With the -9 floor kept: match 50/54, top-1 43, recall@12 0.910. The gap test now encodes the equal-weight tie: a total inversion is broken by retrieval order, the reranker's favourite lands second. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…ons by what agents ask for
Per-lane attribution on the 79-case benchmark (scratch overlay, lexical /
dense / reranker ranks per candidate) traced four top-1 misses to the
function's own first sentence, not to fusion: every fusion knob tried
(RRF K, lexical pruning, dense weight, sibling cap, named-worker anchor)
moved one case up and another down, while the descriptions moved three
cases up and none down.
image_resize::resize "Resize an image via channel I/O" — no width or
height, so browser::resize (which names both) won
shell::fs::ls "List directory contents" — no files/folder/
filesystem, lost to six directory::*::list
storage::putObject "Write a small object inline" — not "upload"; the
cross-encoder tied it with getObject/deleteObject
a2ui::surface::undo "Restore ... from a prior revision" — no undo/edit
Measured against the snapshot with these four sentences and the pipeline
otherwise unchanged: top-1 43 -> 46 of 54, MRR@1 0.796 -> 0.852, nDCG@12
0.834 -> 0.867, no case worse. undo stays 5th: the cross-encoder reads
"roll back the last edit" as a database rollback (-0.33 vs -8.99 logits)
and no honest sentence outvotes that.
Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
… for search
Per-lane attribution on the search benchmark showed the ranking ceiling is
the first sentence of each function's description, so every worker's
descriptions were reviewed against one rule: the first sentence is an
imperative verb plus object in the words an agent uses to ask for the
capability (upload, list files, send a message); mechanics move after it
or go; siblings in a namespace differ within their first five words;
nothing gets longer unless its first sentence was under eight words.
203 descriptions changed across 41 workers, 1,480 characters fewer in
total (MOT-4639's trim stands). Goldens that pin descriptions were
regenerated (browser, github, tailscale, memory, memory-consolidate,
computer, shell, rbac-proxy, security-scan, a2ui). Notable:
gantry::verify shipped with no description at all
email::* said "message" (the Slack word); email::search led with IMAP
github::* first sentences were { req } -> { resp } shape dumps (30)
database::transactionQuery / query and transactionExecute / execute
shared their first five words; rollbackTransaction now says SQL
the coding-agent family (codex, grok, opencode, cursor, pi, claude)
now names its agent and action the same way in run / stop / status
memory::doctor no longer carries "recall"; memory::recall says search
state::* named internal vocabulary ("Get a group from state")
Left alone on purpose: *::on-config-change, metadata.internal functions,
trigger types, console-only / HTTP-ingress surfaces, MCP-compat
duplicates, approval-gate's human-only functions, provider plumbing.
compose::* has no source in this repo. iii-directory/tests/fixtures/
discover_catalog.json is a frozen snapshot and still quotes the old
sentences.
Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
The floor was measured brittle while linting descriptions: one genuine match's reranker leader scored -7.86, -8.82 and -9.46 across three phrasings of its own description, and -9 abstained on the last one. On the 79-case set -10 ranks identically to -9 (same two hard paraphrases still abstain at -10.4 and -11.3, same false positive silenced) while keeping a logit of margin against wording noise. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…iptions after the lint tests/fixtures/discover_catalog.json is the frozen 512-function catalog behind the 38 lexical relevance tests. Its descriptions are refreshed from the lint ledgers (173 of 512), ids and schemas untouched. Four tests failed on the BM25-only lane and exposed two wordings to fix at the source rather than in the tests: state::set's "Store a value under a key in a state scope" echoed one query so exactly that coverage pruning starved state::get on "store ... and read it back" and on "persist ... and read it back later"; it now reads "Store or write a value at a key in a state scope; it persists across sessions", which keeps set and get symmetric for store / write / persist wording. fp::nth's new "Get array element n" rode along on "get the value"; it now says "Return array element n". One test input changed: the session-repeat test overlapped its first query through the old "Set a value in state" wording and now asks "store a value at a key" instead. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…on floor
On 70 held-out queries authored blind from function ids the floor is a
net loss: it abstains on "figure out why a query is slow" (database::explain
was top-1) and "grab a picture of what the page looks like right now"
(browser::screenshot was 2nd) to silence one no-match page ("compose a
haiku"). Two genuine matches per 40 for one silenced page per 15, on top
of the two hard paraphrases it already cost on the 79 tuned cases. Hard
paraphrases score like noise in this cross-encoder (-10 to -11 logits),
so no absolute cut separates "nothing serves this" from "the target is
phrased unlike its description".
Without the floor the branch beats main on every held-out metric but
one extra no-match page: match 34 vs 32 of 40, top-1 23 vs 22, multi
7 vs 5 of 15, recall@12 0.771 vs 0.700, nDCG@12 0.644 vs 0.602.
Abstention stays with the 0.30 cosine admission floor; the relative gap
cut keeps trimming padding under a real leader.
Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…one line `ruff format --check hermes` failed: the lint shortened the description enough to fit on a single line, and ruff removes the now-pointless implicit concatenation. The other three wrapped descriptions in the file still exceed the line limit and stay as they are. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
…rprint `verify_oracle.py --parser-runtime` hashes every git-tracked file under scrapling/src, so the lint's 11 reworded descriptions in scrapling/src/schemas.py made the frozen manifest disagree with the tree and the compat-certification job aborted before running any differential: diff .source.files[7].sha256: frozen='b84a9cab…' current='03950dbb…' diff .source.files[7].size: frozen=19985 current=19947 diff .source.sha256: frozen='1b03fc29…' current='8a584ace…' Recomputed the three affected values with the verifier's own file_record and records_digest over `git ls-files scrapling/src`; they match the "current" values CI reported. Nothing else in the manifest changes: the python packages, parser assets, browser archives and host records are untouched, and `source.version` still tracks scrapling/pyproject.toml. The full `--write` refresh is reserved for an intentional oracle bump and needs the six release archives, so only the source block was re-frozen. Cross-checked with an AST parse of the Python registry that all 19 scrapling-native descriptions still equal their committed goldens. Claude-Session: https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs
andersonleal
force-pushed
the
feat/minilm-followups
branch
from
September 7, 2026 13:16
b44239f to
5245318
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes MOT-4680. Follow-ups carried over from MOT-4657 (#1084), plus the search-quality work they led to.
What this changes
Function descriptions across the repo (the biggest win). Per-lane attribution on the search benchmark showed the ranking ceiling is not the fusion, it is the first sentence of each function's description. 205 descriptions in 41 workers were reviewed against one rule: the first sentence is an imperative verb plus object in the words an agent uses to ask for the capability; mechanics move after it; siblings in a namespace differ within their first five words; nothing gets longer unless its first sentence was under eight words. Net 1,480 characters fewer, so MOT-4639's token trim stands.
gantry::verifyshipped with no description at all;github::*first sentences were{ req } -> { resp }shape dumps; the wholeemailfamily said "message", the Slack word.Ranking knobs, measured not guessed. Reranker weight 1.25 → 1.0, gap 10 → 12, rerank depth 48 → 24 (identical rankings, half the cross-encoder pairs, benchmark wall time 45.6 s → 32.7 s). RRF K was swept 10/20/30/60 and is not a lever, so it stays at 60.
The absolute reranker floor was tried and removed. Added at −9, raised to −10, then dropped: on unseen queries it cost two genuine matches per 40 to silence one no-match page per 15. Abstention stays with the 0.30 cosine admission floor. The relative gap cut still trims padding under a real leader.
approval-gate lockstep.
harness(a dev-dependency) moved to iii-sdk 0.23.0 and approval-gate's lockfile never followed, so every--lockedcargo command there failed before compiling.cargo update -wfixes it; the crate's own pin stays=0.22.1-alpha.25.approval::on-config-changenow carriesmetadata.internal: true.CI proof for MiniLM on Apple Silicon and Windows MSVC.
Cargo.toml's target cfg decides whether ort/fastembed are dependencies andbuild.rs'sMINILM_TARGETSdecides whether the dense lane compiles; a target where they drift builds green as BM25-only. A new matrix job links the static ONNX Runtime on macOS and Windows and fails unlesscfg(minilm)is on.One unrelated fix to unblock CI.
iii-directory's bundled agent is pinned byte-equal toharness/prompts/default.txt; #1085 changed the prompt without regenerating it, socargo test --libis red on main today. Verified on a cleanorigin/mainworktree before fixing.Measured on 70 held-out queries
A sub-agent authored 70 new cases (40 match, 15 multi, 15 no-match, 34 namespaces) blind, from function ids and request-parameter docs only, never a description, so the phrasing owes nothing to either the old or the new wording. Same harness, same catalog, one variable at a time:
The descriptions generalize: at a fixed pipeline they are worth +2 match, +1 top-1 and +0.05 recall on queries nobody tuned for. The knobs are neutral on quality here and keep their latency win. The branch beats main on every metric except one extra no-match page.
On the 79 tuned cases the same changes give top-1 43 → 47 of 54, MRR@1 0.796 → 0.870, nDCG@12 0.834 → 0.885.
Verification
iii-directory: 434 tests, clippy--all-features --all-targets, fmt.cargo fmt --checkon every touched crate;python3 -m py_compileon the touched Python; biome on the touched TypeScript.discover_catalog.jsonwas refreshed so the 38 lexical tests run over the shipped wording. That run caught two rewrites that were actively harmful and they were fixed at the source:state::setechoed a query so exactly that coverage pruning starvedstate::get, andfp::nth's "Get array element" rode along on "get the value".Open, not in this PR
editor::git::hunksfor "what I changed since the last commit",worktree::removefor "throw away a scratch checkout",memory::save/memory::recallfor "remember that I prefer" / "dig up what I told you",canvas::createfor "draw a flowchart",tailscale::serve::addfor "reachable from my other machines".slack/src/surface.rsandtelegram-bot/src/surface.rscarry a second, divergent description set consumed only by their schema goldens.compose::*descriptions live in the engine daemon, not this repo.https://claude.ai/code/session_01CSg2hqn9zHWvYpWXkbKEVs