Repository navigation
Add MCP documentation search and page tools on the website's search API - #472
Conversation
tim-kos
left a comment
There was a problem hiding this comment.
Adversarial code/product review: two major findings.
[major] The advertised API/FAQ scopes search the wrong index and can return no useful results.
Location: packages/mcp-server/src/docs.ts:16 and packages/mcp-server/src/docs.ts:81.
Problem: every search fetches only the root /llms.txt, then matchesScope() filters those root entries by /docs/{scope}/. The root index is primarily a Robot/overview index and does not contain the exhaustive API or FAQ pages that the scope enum promises. The scoped indexes added by content#5711 are never fetched.
Likely consequence: valid calls such as an API-scoped search for authentication or an FAQ-scoped troubleshooting query return an empty/partial result even though the relevant public pages exist.
Direct evidence: docsIndexUrl is a single constant used at line 89 for every scope; scope changes only the post-parse path filter. The unit test supplies an artificial root index containing API data, so it does not exercise production index topology.
Required change: select /docs/{scope}/llms.txt for scoped searches. Define an explicit strategy for all (for example, aggregate the bounded product indexes) and test API/FAQ routing. If deployment ordering matters, handle a missing scoped index deliberately rather than silently presenting an empty search surface.
[major] max_chars bounds tool output only after the entire upstream body is buffered.
Location: packages/mcp-server/src/docs.ts:37 and packages/mcp-server/src/docs.ts:112.
Problem: fetchDocsText() calls response.text(), which reads the complete response into memory. Only afterward does getTransloaditDoc() slice to maxChars.
Likely consequence: a mistakenly huge or compromised allowed-origin response can consume unbounded memory/network despite the README and JSDoc describing the result as bounded. The origin allowlist prevents SSRF but does not impose a response-size boundary.
Direct evidence: lines 45 and 112–118 show full buffering followed by slicing; there is no Content-Length check or streamed byte/character cutoff.
Required change: read the response stream with a hard upper bound (and cancel once reached), preserving enough information to set truncated. Add an oversized streamed-response test. Also reject unexpected content types so an HTML error page is not returned as Markdown.
Scores (1–5): factual accuracy 4; clarity 4; completeness 2; source grounding 4; product accuracy 3; AI-agent readability 3.
tim-kos
left a comment
There was a problem hiding this comment.
Review follow-up: addressed in eb8a663. Search now uses the scoped product indexes with a root-index rollout fallback. Documentation bodies are streamed to a hard character bound, oversized indexes fail, and unexpected content types are rejected. The MCP workspace check passes with 4 suites / 9 tests.
kvz
left a comment
There was a problem hiding this comment.
There already are:
transloadit_list_robotstransloadit_get_robot_help
Do we really need
transloadit_search_docs(read-only; no account credentials required)transloadit_get_doc(read-only; no account credentials required)
? If so, can they be more in line with existing names or vice versa?
|
@kvz Addressed the overlap and naming feedback:
Full |
kvz
left a comment
There was a problem hiding this comment.
Thanks for picking this back up, Tim. Here is where I'd like to take it, and why.
What we'll do
- Content gets one small JSON endpoint,
GET /api/docs/search, on top of the Algolia index that the site search and the WebMCPsearch_docstool already use. It returns{results: [{title, url, markdown_url, excerpt, section}]}, English only, with a contract test in Content. - This PR becomes two thin tools on top of that endpoint and the existing
.mdpages:transloadit_search_docs(query, section?, limit?)andtransloadit_get_doc(url, start_index?, max_chars?). Your bounded streaming fetch, the origin restriction and the tests stay. - Smaller changes while we're there: paginate page reads with a default of about 15,000 characters, follow same-origin redirects and accept
www.URLs, give a missing page its own not-found error and log failures instead of a barecatch, returnnext_steps(a Robot page hit points totransloadit_get_robot_help), and dropdocs/prompts/2026-11-12-mcp-docs-review.mdfrom the PR.
Why
- Parsing
llms.txtwith a regex at runtime turns a prose format for models into an API. Nothing in Content tests that line format, so a change there would break the tool in production, and today every failure reads as "temporarily unavailable". - Search is the one thing agents can't already do with a web fetch, and substring matching on titles and one-line descriptions is the weakest way to offer it. The site already has full-text search over headings and body text.
- Search logic inside the npm package can only improve through a release, an API2 bump and a rollout. Behind a Content endpoint it improves with a Content merge, and we could even swap the search provider without touching the MCP server.
- On my September question: yes, I think we need search. ChatGPT and Gemini connectors often have no web access, and no existing tool offers it. On naming:
search_docsdoes relevance search, whilelist_robotsenumerates a catalog with a cursor, so I'd rather have honest verbs than mirrored ones. The Robot tools stay as they are. - Tool names in a directory-listed server are hard to change once they ship, so I'd like the shape right first.
How
I'll land the Content endpoint first, then rework this branch on top of it and ping you when it's ready for a look. No need to change anything here in the meantime.
Why: Agents connected to the MCP server can look up Robots and their parameters, but they cannot search or read the rest of the documentation: API authentication and endpoints, SDK setup, FAQs and integration guides. ChatGPT and Gemini connectors often have no web access of their own, so this has to come from the server.
What: Two read-only tools on top of what the website already serves, per Kevin's review:
transloadit_search_docsquery, optionalsection,limit(1–20, default 8)title,url,markdown_url,excerptandsectiontransloadit_get_docurl(a/docs/path or a transloadit.com URL),start_index,max_chars(default 15,000)next_start_indexto continueGET https://transloadit.com/api/docs/search/(transloadit/content#7469), a versioned JSON contract on the site's full-text search. Search quality can now improve with a Content merge, without an MCP server release. The earlier approach in this PR parsed thellms.txtindexes with a regex and matched titles only..mdpages, paginate by Unicode code point, follow at most three redirects that stay underhttps://transloadit.com/docs/, and acceptwww.andhttp://URLs.mcp_invalid_docs_query,mcp_invalid_docs_url,mcp_doc_not_foundandmcp_docs_unavailable. Failures are logged by category only, never with URLs or bodies.next_steps: search points to the page tool, and Robot pages also point to the structured Robot help.Validation
c41666d7.yarn checkpasses.mainis merged in; force pushes are not allowed on this branch, so the history keeps the earlier commits.🤖 Claimed by kvz on
studio1/node-sdksince 2026-11-17 · resume:claude --resume 26eaa88e-b3fa-4cc1-bcce-0c7ff5b6bab7