experiment: evaluate capability discovery and agent search behavior

[reachjalil]

Caution

Experiment only — do not merge. This draft combines three independent PRs so their interaction can be evaluated. Findings should be applied back to the source PRs; this integration PR is not intended to become the delivery branch.

Goal

Test one end-to-end hypothesis: capability discovery should find the right tool with fewer agent searches and less repeated external-MCP work, without changing execution, authorization, schemas, or connection recovery behavior.

The three changes address different layers of the same user journey:

1. The prompt decides when and how often the agent searches.
2. Ranking decides which results appear first.
3. Caching decides how much remote discovery work repeated searches require.

Evaluating any layer alone can give a misleading result. Faster search is less valuable if the prompt mandates unnecessary calls; better ranking is less valuable if the agent always issues several variants; a leaner prompt is risky unless result relevance and recovery behavior remain reliable.

Included source PRs

PR	Exact head	Role in the experiment
#2966 — Lexical capability ranking	8e68afb406672c6c9b854a6fc42729a5d8f7e60a	Globally reranks native, REST, admin, external MCP, marketplace, and skill matches using normalized lexical relevance while preserving current result contracts.
#2967 — External MCP search cache	6d3bcbf43b51e0e626b3eef0a4e102fda4b38bae	Reuses complete sanitized external tool catalogs for 60 seconds during search, with failure exclusion, concurrent-miss coalescing, credential-aware keys, and bounded LRU memory. Execution stays live.
#2977 — Agent instruction optimization	bca13ef1b114a415d804981307e8c3094715847d	Removes mandatory 2-4 search variants and reduces duplicated always-on instructions. The agent starts with one focused query and refines only when no result is relevant.

This branch started from dev at 6bee537e7, merged each source head with a separate provenance-preserving merge commit, and is now synchronized through dev commit a314ea96d. The sync also brought in #2965's runtime Connect skill catalog; its instruction was reviewed under the same prompt methodology and reduced to a short heading, retrieval rule, and prompt-injection trust boundary without changing catalog data or retrieval behavior.

Expected combined behavior

For a connected-service request:

1. The agent starts with one concise capability query derived from the task.
2. Search loads the current native sources and the authorized external MCP catalogs.
3. External catalogs may come from the bounded search-only cache, but live authorization and credential checks still run.
4. Results are reranked across sources; actionable connection status remains ahead of unrelated callable tools.
5. When a relevant result exists, the agent stops searching and executes its exact returned name.
6. When results are empty or irrelevant, the agent retries with meaningfully broader or different keywords rather than a fixed query quota.
7. Execution performs live schema discovery and permission checks; the cache never authorizes or executes a capability.
8. A downstream connector failure is reported using the returned connection action and is not misdiagnosed as an OpenWork Cloud reconnection problem.

Why this is an experiment

• Lexical heuristics can improve common queries while producing regressions on ambiguous or provider-specific vocabulary.
• A shorter prompt may work differently across OpenAI, Anthropic, and local/other supported models.
• A process-local cache improves warm discovery but does not improve cold starts and must not conceal credential or catalog changes.
• Tool-call count and latency only count as improvements if the final capability, arguments, and user-facing recovery path remain correct.

This branch exists to collect that evidence before deciding whether each source PR is valuable independently, needs revision, or should be closed.

Success criteria

• Relevant capability is normally found with one search.
• A second search happens only after an empty or irrelevant first result and uses materially different keywords.
• Exact intended capabilities rank first for representative native and external workflows.
• Warm repeated external searches issue no additional remote catalog requests within the cache window.
• Search results preserve bodySchema, argumentsSchema, schemaDigest, invocation metadata, marketplace requirements, and connection actions.
• Authorization, tenant/member scope, credential validation, and execution remain live.
• Search-call count and elapsed time improve without reducing task completion or recovery quality.
• Behavior is acceptably consistent across at least one OpenAI, one Anthropic, and one supported local/other model.

Combined validation checklist

Composition and automated contracts

• Merge all three exact source heads and synchronize the integration branch through current dev (a314ea96d), resolving prompt/Connect-skill conflicts intentionally.
• Install locked dependencies and pass the isolated Hub readiness check.
• Review the combined diff against current dev: 21 files, limited to the three source PRs plus the small Connect skill-instruction adaptation required by the dev sync.
• Run the ranking, capability-search, Google Workspace, external-cache, focused external-search divergence, timeout, instruction, knowledge, steering, extension-preview, Connect skill catalog, and runtime-config tests (99 focused tests passed).
• Run Den API and OpenWork server typechecks.
• Confirm the warm benchmark performs zero extra remote catalog requests while returning the same schema-bearing matches (4 cold remote requests, 0 warm).
• Confirm failure exclusion, miss coalescing, TTL expiry, entry bounds, and byte bounds.
• Exercise credential and connection revision invalidation explicitly in the combined branch.

Search and execution journey

• Test representative list/create, Gmail, Calendar, Drive, Memory Bank, marketplace, skill, and external MCP queries.
• Confirm a relevant first result stops further searching.
• Confirm an irrelevant first result triggers a meaningfully refined query.
• Confirm synonymous queries reuse external discovery without distorting ranking.
• Confirm execution uses only exact returned names and remains fully live.
• Compare search calls, remote catalog requests, latency, and task success against current dev.

Contracts, recovery, and prompt routing

• Confirm all native and external schema/invocation fields are preserved.
• Confirm connection-status ordering and exact action relay.
• Confirm connector failures do not cause incorrect OpenWork Cloud reconnect advice.
• Confirm product questions use bundled docs while service actions use live capabilities.
• Confirm OpenWork UI, external browser, past-session, and Memory Bank routing still follows the correct tool boundary.
• Confirm Memory Bank writes remain explicit, confirmed, and free of secrets or sensitive personal data.
• Run behavioral trials across OpenAI, Anthropic, and a supported local/other provider.

Decision gate

• Record regressions, provider differences, call counts, latency, and lost behavior.
• Apply required fixes to the appropriate source PR rather than treating this branch as the final implementation.
• Decide independently whether feat(mcp): rank capability search lexically #2966, perf(mcp): cache external tool discovery for search #2967, and refactor(server): optimize injected agent instructions #2977 should proceed, change, or close.
• Close this integration PR after the experiment. Do not merge it.

Current validation

• Hub dependency preparation — passed.
• Hub environment/readiness check — passed.
• All three source heads are preserved as merge commits; latest dev sync conflicts were resolved intentionally.
• Combined focused tests — 99 passed across server prompt/steering, Connect skill injection, ranking/search, Google Workspace fixtures, timeout contracts, cache behavior, and the warm-cache integration path.
• Warm-cache benchmark — 4 remote catalog requests on the cold search sequence, 0 on the warm sequence.
• Combined Den/server typechecks — passed.
• Provider-level behavioral trials — not run yet.
• Visual evidence — not applicable to branch composition; observable agent trials remain pending.

Related experiment-status note and checklist: #2977 discussion.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openwork-landing	Ready	Preview, Comment, Open in v0	Jul 22, 2026 7:30am

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
openwork-diagnostics	Skipped		Jul 22, 2026 7:30am

[vercel]

@reachjalil is attempting to deploy a commit to the Different AI Team on Vercel.

A member of the Team first needs to authorize it.

[reachjalil]

Experiment status: this draft is an integration harness for #2966, #2967, and #2977. It is intentionally not intended to be merged. Please use the validation checklist and record findings here; any production-worthy fixes should be applied to the appropriate source PR, then this combined PR should be closed.