[daud99]

Problem

The CommonCrawl source returns no URLs at all.

Root cause: The CDX API returns raw CDX-format lines by default:

com,example)/ 20260508... {"url": "http://www.example.com/", ...}

But the regex extractor in pkg/extractor/regex_extractor.go is compiled with ^ and $ anchors, so it only matches when the entire input line is a URL. A CDX record line never matches, so every line silently returns zero results.

Fix

Add &output=text&fl=url&collapse=urlkey to the CDX query URL, which makes the API return one plain URL per line — the same approach already used by the Wayback Archive source (&output=txt&fl=original).

-currentSearchURL := fmt.Sprintf("%s?url=*.%s", searchURL, rootURL)
+currentSearchURL := fmt.Sprintf("%s?url=*.%s&output=text&fl=url&collapse=urlkey", searchURL, rootURL)

• output=text&fl=url — return only the URL field, one per line
• collapse=urlkey — deduplicate entries where the same URL was crawled multiple times

Testing

Tested against example.com — before the fix: 0 results; after: thousands of URLs returned across all subdomains.

Summary by CodeRabbit

• Bug Fixes
  • Improved Common Crawl search query formatting to produce more consistent, better-structured results.
  • Added robust handling for invalid search URLs so failed parses now surface errors and stop processing rather than continuing with invalid requests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5da3486b-2bcc-486e-b161-a5c7036bd14a

📥 Commits

Reviewing files that changed from the base of the PR and between b8c9581 and 1fa40d4.

📒 Files selected for processing (1)

• pkg/source/commoncrawl/commoncrawl.go

💤 Files with no reviewable changes (1)

• pkg/source/commoncrawl/commoncrawl.go

---

Walkthrough

getURLs refactors Common Crawl search URL construction from fmt.Sprintf to url.Parse with explicit query parameters (url=*.<rootURL>, output=text, fl=url, collapse=urlkey). The function emits an error and returns early on parse failure, uses the encoded URL as currentSearchURL, and removes the unused fmt import and triple-encoded URL normalization logic.

Changes

Common Crawl API query parameters

Layer / File(s)	Summary
Search URL parsing and query assembly pkg/source/commoncrawl/commoncrawl.go	getURLs now parses searchURL and sets query parameters using url.Query() methods instead of fmt.Sprintf, encodes the result into currentSearchURL, and returns false with an error on parse failure. Removes unused fmt import and eliminates the extracted-URL normalization that handled triple-encoded cases.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I parsed the search and set each query key,
Four params dancing: url, text, and key,
No triple tricks, no sprintf strings—
Just tidy URL-building things! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title clearly and concisely describes the main fix: adding output=text&fl=url parameters to the CDX query so URLs are returned.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/source/commoncrawl/commoncrawl.go`:
- Line 124: The code builds currentSearchURL by interpolating rootURL into a
query string which can produce malformed queries if rootURL contains reserved
characters; instead parse searchURL with net/url, create/modify its Query()
(url.Values), set the "url" param to "*."+rootURL (or to a sanitized hostname if
you prefer), set "output"="text", "fl"="url", "collapse"="urlkey", then assign
currentSearchURL = u.String(); update the code around the currentSearchURL
construction (use searchURL, rootURL and the currentSearchURL variable) so
parameters are safely URL-encoded.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6c79657a-bd0c-45ac-b942-575f595dbce3

📥 Commits

Reviewing files that changed from the base of the PR and between 96031ff and 0b82e8a.

📒 Files selected for processing (1)

• pkg/source/commoncrawl/commoncrawl.go