feat(crawl): add zenrows crawl for the Crawl API

[StefanoGuerrini]

What this adds

A zenrows crawl command for the Crawl API (https://api.zenrows.com/v1/crawls). Crawl is in beta. Crawl takes one start URL, follows the links on each page up to depth on the start URL's registrable domain (subdomains count), and returns the URLs it keeps, optionally with each page's HTML.

zenrows crawl start <url> --depth N [--max-items N] [--max-pages N] [--include P]... [--exclude P]...
                         [--html] [--wait] [--timeout ms] [--idempotency-key K] [--no-signup]
zenrows crawl status <id>
zenrows crawl wait <id> [--timeout ms]
zenrows crawl results <id> [--out file] [--cursor C --limit N] [--download]
zenrows crawl content <id> <content_id|content_url> [--out file]
zenrows crawl list [--limit N] [--cursor C]
zenrows crawl stop <id>

Every subcommand prints JSON with --json, in the same {ok, ..., error} shape the other commands use. results --out with --json prints the envelope with file instead of results.

• src/core/crawl-api.ts: the client. createCrawl, getCrawl, listCrawls, stopCrawl, getCrawlContent, downloadCrawl (returns the X-Crawl-Status status with the lines), listAllResults (follows next_cursor; on a running crawl it returns what is kept so far and ends at the first empty page, with partial: true), waitForCrawl (polls with limit=1, 2s backoff x1.5 capped at 15s, 600 s default; on timeout the crawl keeps running). Auth is the X-API-Key header. ZENROWS_CRAWL_API_BASE overrides the base; otherwise it follows the configured apiBase. No Crawl call is retried.
• create sends only the fields the caller set. --html sets output_format: "html". Without it, the crawl returns URLs only. With it, each kept page is one more fetch that --max-pages counts.
• start --wait and wait poll until the crawl ends. Ctrl-C stops the wait, not the crawl: the CLI says the crawl is still running, prints zenrows crawl wait <id> and exits 130. --follow is gone.
• stop on a crawl that already ended prints the status it ended with, not "Stopped".
• Error codes: CRAWL_NOT_ENABLED (403 REQS008), CRAWL_INVALID_REQUEST (400 and 422; for idempotency_key_reused the advice is to send a new key or no key), CRAWL_REQUEST_IN_FLIGHT (409, retry once the first request ends), CRAWL_NOT_FOUND and CRAWL_CONTENT_NOT_FOUND (404), CRAWL_TOO_MANY_CRAWLS (429: the account has reached its limit of active jobs, 3 by default, shared with its Batch jobs; structured retry_after in seconds), CRAWL_WAIT_TIMEOUT and CRAWL_FAILED (a crawl that ended failed, any other 403, or another error). 401 maps to AUTH_INVALID. 402 goes through the existing KEY_CREDIT_CAP_REACHED and POLICY_MAX_CREDITS_EXCEEDED errors. Every error carries the API's code as server_code when the API sent one, and crawl_id once the crawl exists.
• start checks the domain policy and max_pages_per_run (default 1000) before any network call, and writes a run artifact. The help says that this local policy caps --max-pages.
• The crawl capability (beta), a crawl skill (beta) with its registry entry, crawl lines in the zenrows and trace-debug skills, a README section and table row, and a docs/capabilities.md row. The help, README and skills say Beta.

Test guards changed on purpose

tests/production-surface.test.ts asserted that crawl is an unknown command, and tests/registry.test.ts asserted that there is no crawl skill. Both now check only discover and monitor, and also check that crawl is registered. tests/capabilities.test.ts pins the crawl status to beta.

Tests

• npm run typecheck: clean.
• npm test: 327 passed, 0 failed. New: tests/crawl-api.test.ts covers request shape, response parsing, error mapping (403 REQS008 and other 403, 404 both codes, 400, 409, 422 including idempotency_key_reused, 429 with retry_after and no retry, 401, 500, a body without a code), server_code and crawl_id on errors, paging on ended and running crawls, and waiter termination and timeout. tests/crawl-command.test.ts covers the command, including that undeclared flags (and --follow) are rejected before any request, partial results, the --out --json envelope, stop on an ended crawl and Ctrl-C during a wait.
• E2E, tests/e2e/crawl.e2e.test.ts: drives the built CLI binary. It runs only with npm run test:e2e, and it skips unless ZENROWS_E2E=1, ZENROWS_API_KEY and ZENROWS_E2E_CRAWL_URL are set. ZENROWS_E2E_CRAWL_INCLUDE and ZENROWS_CRAWL_API_BASE are optional. docs/contributing.md has a "Running the Crawl e2e test" section.

export ZENROWS_API_KEY="$(tr -d '\n' < path/to/api.key)"
ZENROWS_E2E=1 ZENROWS_CRAWL_API_BASE=$ZENROWS_CRAWL_API_BASE \
  ZENROWS_E2E_CRAWL_URL=<start-url> ZENROWS_E2E_CRAWL_INCLUDE=<pattern> npm run test:e2e

I ran it against a live ZenRows API deployment on the first commit, with the target set through ZENROWS_E2E_CRAWL_URL: 1 passed, 0 failed. The run started a crawl and waited for it to complete. Every result matched the include pattern. It read one page's HTML, and the NDJSON export had one line per result. The crawl appeared in the list, stop on the ended crawl returned its terminal status, and an unknown id gave CRAWL_NOT_FOUND. The e2e test now uses --wait and CRAWL_TOO_MANY_CRAWLS; it did not run again after the fix commit.

[StefanoGuerrini]

Crawl Beta public-surface review

This review checks the public CLI surface against what production serves to a crawl-beta customer today: Crawl 475016a (docs/openapi/api.yaml) and gateway e382348. crawl-beta is at 100%. crawl-beta-extract (output_format: json, discovery: pagination) is staff-only. The public surface is the command, subcommands, flags, --json shapes, error codes, exit codes, env vars, the registry entries and the skill text. Reviewed at 7be627f: npm run typecheck is clean, and npm test passes 320 of 320 tests. The e2e test needs a live key and did not run.

Verdict: changes needed. Items 1 to 6 become breaking changes if they ship as they are.

Fix before merge: breaking if shipped

1. No Beta marking. The capability and the skill have status available (registry/capabilities.json:34, registry/skills.json:51), and the tests pin it (tests/capabilities.test.ts:14). Batch uses beta. The README table (README.md:42) and the master skill table (skills/zenrows/SKILL.md:77) also say "available". Set both statuses to beta, and say "Beta" in crawl --help, the README and the crawl skill (proposed decision 1). Agents read this status, so a later downgrade from available to beta is a contract change.
2. 429 maps to CRAWL_QUOTA_EXCEEDED (src/core/crawl-api.ts:271). The name says quota, but too_many_crawls is the account's active-job limit. Retry-After goes only into likely_cause text (:269). Rename to CRAWL_TOO_MANY_CRAWLS and add a retry_after field (proposed decision 4). Agents branch on the code, so a rename later breaks them.
3. 400, 409 and 422 all map to CRAWL_INVALID (src/core/crawl-api.ts:289). A 409 (idempotency_request_in_flight) is safe to retry; a 422 is not. One code makes an agent unable to tell them apart. Use CRAWL_INVALID_REQUEST for 400 and 422 (including idempotency_key_reused) and CRAWL_REQUEST_IN_FLIGHT for 409 (proposed decision 4).
4. 404 content_not_found maps to CRAWL_NOT_FOUND (src/core/crawl-api.ts:257). Only the message differs. Use CRAWL_CONTENT_NOT_FOUND (proposed decision 4).
5. Wait timeout is CRAWL_TIMEOUT (src/core/crawl-api.ts:449, skills/crawl/SKILL.md:38). Rename to CRAWL_WAIT_TIMEOUT (proposed decision 4). Also update the ErrorCode union (src/core/errors.ts:43).
6. start --follow with alias --wait (src/cli/commands/crawl.ts:114, help :54). Keep --wait only (proposed decision 10). The README (:172), the crawl skill (:25, :36, :43) and the master skill (skills/zenrows/SKILL.md:38) teach --follow, so scripts will use it. Removing it later breaks them.

Fix before merge: behaviour and docs

7. The server code is not a field. ToolkitError has no field for it (src/core/errors.ts:51). The server code appears only inside the likely_cause text (src/core/crawl-api.ts:233). Add it as a field, for example server_code (proposed decision 4).
8. results refuses a running crawl with INVALID_USAGE (src/core/crawl-api.ts:414). Return what is kept with partial: true, as results --download already does (src/cli/commands/crawl.ts:252) (proposed decision 2).
9. The 422 idempotency_key_reused advice is wrong. It gets "Fix the reported parameter and retry" (src/core/crawl-api.ts:297). A retry with the same key fails again. Tell the agent to send a new key, or no key.
10. The 429 text has the wrong meaning. The capability notes (registry/capabilities.json:38), the README (:185), the crawl skill (:67) and the trace-debug skill (skills/trace-debug/SKILL.md:50) say "too many crawls running". The limit is the account's active Batch jobs: 3 by default, shared with its Batch jobs. Say so. The skill's "about 30s" is correct.
11. --max-pages help says 1 to 100000 (src/cli/commands/crawl.ts:50). The default local policy refuses more than 1000 with POLICY_LIMIT_EXCEEDED before any request (src/cli/commands/crawl.ts:162, src/core/policy.ts:11). This is the same local cap Batch uses, not a production limit. Say in the help and the skill that policy.max_pages_per_run caps it locally.

Matches production

• start sends url and depth (required) and only the flags set. --html sets output_format: "html". No flag sets json or discovery.
• Flag ranges match: depth, max_items, max_pages 1 to 100000; results --limit up to 10000; list --limit up to 100. The help calls the defaults of 10 API defaults.
• 403 REQS008 maps to CRAWL_NOT_ENABLED, 401 to AUTH_INVALID, 402 to the existing KEY_CREDIT_CAP_REACHED and POLICY_MAX_CREDITS_EXCEEDED. Other statuses, including a 403 without REQS008, map to CRAWL_FAILED.
• --json passes the whole crawl object through, so discovery, extract_calls and unknown fields reach the agent. Enums are open.
• results --download reports crawlStatus from X-Crawl-Status and partial (proposed decision 3).
• list --json prints next_cursor: null when the API leaves it out on the last page.
• A failed crawl exits 1 with CRAWL_FAILED; stopped exits 0. Every error exits 1, as in all other commands (src/cli/index.ts:98).
• ZENROWS_CRAWL_API_BASE stays (proposed decision 9). Auth is the X-API-Key header.
• start auto-creates a Free plan account through ensureApiKey unless --no-signup is set, as fetch does. crawl-beta is at 100%, so the new account can crawl.
• The guard tests drop crawl from the "removed non-GA" lists and still guard discover and monitor.

Can wait

These are additive and need no change now:

• The wait default is 600 s (src/core/crawl-api.ts:441). All surfaces will use one value; we have not picked it yet (proposed decision 8).
• No flag for output_format: json or discovery. A new flag can add them.
• Errors do not expose the RateLimit headers.
• crawlFailure has no specific advice for domain_not_allowed or internal_error; the generic advice applies.

Release: minor bump (1.5.0), since this adds a command.

[StefanoGuerrini]

Replaced by #31: the same code, with a history that drops the AI co-author trailer. The review on this PR still applies.