Skip to content

Add MCP documentation search and page tools on the website's search API - #472

Merged
kvz merged 10 commits into
mainfrom
agent/mcp-docs-retrieval
Nov 17, 2026
Merged

kvz merged 10 commits into
mainfrom
agent/mcp-docs-retrieval

Conversation

@tim-kos

@tim-kos tim-kos commented Aug 18, 2026 •

Copy link
Copy Markdown
Member

Why: Agents connected to the MCP server can look up Robots and their parameters, but they cannot search or read the rest of the documentation: API authentication and endpoints, SDK setup, FAQs and integration guides. ChatGPT and Gemini connectors often have no web access of their own, so this has to come from the server.

What: Two read-only tools on top of what the website already serves, per Kevin's review:

Tool Input Returns
transloadit_search_docs query, optional section, limit (1–20, default 8) Ranked results with title, url, markdown_url, excerpt and section
transloadit_get_doc url (a /docs/ path or a transloadit.com URL), start_index, max_chars (default 15,000) One page as Markdown, with next_start_index to continue
  • Search calls the website's GET https://transloadit.com/api/docs/search/ (transloadit/content#7469), a versioned JSON contract on the site's full-text search. Search quality can now improve with a Content merge, without an MCP server release. The earlier approach in this PR parsed the llms.txt indexes with a regex and matched titles only.
  • Page reads stream the existing .md pages, paginate by Unicode code point, follow at most three redirects that stay under https://transloadit.com/docs/, and accept www. and http:// URLs.
  • Errors are distinct and sanitized: mcp_invalid_docs_query, mcp_invalid_docs_url, mcp_doc_not_found and mcp_docs_unavailable. Failures are logged by category only, never with URLs or bodies.
  • Results return next_steps: search points to the page tool, and Robot pages also point to the structured Robot help.
  • The Robot tools keep their names and behavior. Both new tools are public: no credentials in caller-credentials mode, and the existing OAuth policy on hosted deployments.

Validation

  • CI: all 14 checks pass on c41666d7.
  • 399 MCP unit tests, including search mapping, pagination across two calls, redirects in and out of the docs, 404s and content types; yarn check passes.
  • Live against production on 2026-11-17: "webhook signature" returns the Webhooks "Verify the signature" section first, and the Webhooks page reads back as Markdown in two pages.
  • main is merged in; force pushes are not allowed on this branch, so the history keeps the earlier commits.

🤖 Claimed by kvz on studio1/node-sdk since 2026-11-17 · resume: claude --resume 26eaa88e-b3fa-4cc1-bcce-0c7ff5b6bab7

@tim-kos tim-kos self-assigned this Aug 18, 2026

@tim-kos tim-kos left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial code/product review: two major findings.

[major] The advertised API/FAQ scopes search the wrong index and can return no useful results.

Location: packages/mcp-server/src/docs.ts:16 and packages/mcp-server/src/docs.ts:81.

Problem: every search fetches only the root /llms.txt, then matchesScope() filters those root entries by /docs/{scope}/. The root index is primarily a Robot/overview index and does not contain the exhaustive API or FAQ pages that the scope enum promises. The scoped indexes added by content#5711 are never fetched.

Likely consequence: valid calls such as an API-scoped search for authentication or an FAQ-scoped troubleshooting query return an empty/partial result even though the relevant public pages exist.

Direct evidence: docsIndexUrl is a single constant used at line 89 for every scope; scope changes only the post-parse path filter. The unit test supplies an artificial root index containing API data, so it does not exercise production index topology.

Required change: select /docs/{scope}/llms.txt for scoped searches. Define an explicit strategy for all (for example, aggregate the bounded product indexes) and test API/FAQ routing. If deployment ordering matters, handle a missing scoped index deliberately rather than silently presenting an empty search surface.

[major] max_chars bounds tool output only after the entire upstream body is buffered.

Location: packages/mcp-server/src/docs.ts:37 and packages/mcp-server/src/docs.ts:112.

Problem: fetchDocsText() calls response.text(), which reads the complete response into memory. Only afterward does getTransloaditDoc() slice to maxChars.

Likely consequence: a mistakenly huge or compromised allowed-origin response can consume unbounded memory/network despite the README and JSDoc describing the result as bounded. The origin allowlist prevents SSRF but does not impose a response-size boundary.

Direct evidence: lines 45 and 112–118 show full buffering followed by slicing; there is no Content-Length check or streamed byte/character cutoff.

Required change: read the response stream with a hard upper bound (and cancel once reached), preserving enough information to set truncated. Add an oversized streamed-response test. Also reject unexpected content types so an HTML error page is not returned as Markdown.

Scores (1–5): factual accuracy 4; clarity 4; completeness 2; source grounding 4; product accuracy 3; AI-agent readability 3.

@tim-kos tim-kos left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review follow-up: addressed in eb8a663. Search now uses the scoped product indexes with a root-index rollout fallback. Documentation bodies are streamed to a hard character bound, oversized indexes fail, and unexpected content types are rejected. The MCP workspace check passes with 4 suites / 9 tests.

@kvz kvz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There already are:

  • transloadit_list_robots
  • transloadit_get_robot_help

Do we really need

  • transloadit_search_docs (read-only; no account credentials required)
  • transloadit_get_doc (read-only; no account credentials required)

? If so, can they be more in line with existing names or vice versa?

@tim-kos tim-kos changed the title Add MCP documentation retrieval tools Add MCP documentation discovery and help alongside Robot tools Nov 12, 2026
@tim-kos

tim-kos commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@kvz Addressed the overlap and naming feedback:

  • Renamed the new tools to transloadit_list_docs and transloadit_get_doc_help, matching transloadit_list_robots / transloadit_get_robot_help. Discovery now uses optional search, like list_robots.
  • Kept the Robot tools unchanged and documented the difference in the README and tool descriptions: structured Robot Step parameters versus public API, SDK, FAQ, integration and full Robot prose.
  • Rebased onto current main and preserved the registry/auth metadata.
  • Validated today’s live indexes in every scope. That found and fixed duplicate search results, missing overview links and multi-term ranking. Missing scoped indexes now report an error instead of silently using an incomplete root fallback.

Full yarn check, all 363 MCP unit tests and both stdio tests pass. Live API/FAQ/SDK/Robot/all searches found their expected first page, bounded page retrieval worked, and a real MCP client fetched the authentication guide without account credentials. The PR description and committed review checklist contain the full evidence and the unavailable independent-review tooling limitation.

@tim-kos
tim-kos requested a review from kvz November 12, 2026 14:11

@kvz kvz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this back up, Tim. Here is where I'd like to take it, and why.

What we'll do

  1. Content gets one small JSON endpoint, GET /api/docs/search, on top of the Algolia index that the site search and the WebMCP search_docs tool already use. It returns {results: [{title, url, markdown_url, excerpt, section}]}, English only, with a contract test in Content.
  2. This PR becomes two thin tools on top of that endpoint and the existing .md pages: transloadit_search_docs(query, section?, limit?) and transloadit_get_doc(url, start_index?, max_chars?). Your bounded streaming fetch, the origin restriction and the tests stay.
  3. Smaller changes while we're there: paginate page reads with a default of about 15,000 characters, follow same-origin redirects and accept www. URLs, give a missing page its own not-found error and log failures instead of a bare catch, return next_steps (a Robot page hit points to transloadit_get_robot_help), and drop docs/prompts/2026-11-12-mcp-docs-review.md from the PR.

Why

  • Parsing llms.txt with a regex at runtime turns a prose format for models into an API. Nothing in Content tests that line format, so a change there would break the tool in production, and today every failure reads as "temporarily unavailable".
  • Search is the one thing agents can't already do with a web fetch, and substring matching on titles and one-line descriptions is the weakest way to offer it. The site already has full-text search over headings and body text.
  • Search logic inside the npm package can only improve through a release, an API2 bump and a rollout. Behind a Content endpoint it improves with a Content merge, and we could even swap the search provider without touching the MCP server.
  • On my September question: yes, I think we need search. ChatGPT and Gemini connectors often have no web access, and no existing tool offers it. On naming: search_docs does relevance search, while list_robots enumerates a catalog with a cursor, so I'd rather have honest verbs than mirrored ones. The Robot tools stay as they are.
  • Tool names in a directory-listed server are hard to change once they ship, so I'd like the shape right first.

How

I'll land the Content endpoint first, then rework this branch on top of it and ping you when it's ready for a look. No need to change anything here in the meantime.

@kvz kvz changed the title Add MCP documentation discovery and help alongside Robot tools Add MCP documentation search and page tools on the website's search API Nov 17, 2026
@kvz
kvz merged commit c23589c into main Nov 17, 2026
17 checks passed
@kvz
kvz deleted the agent/mcp-docs-retrieval branch November 17, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants