[dogancanbakir]

Closes #133

Two unrelated causes behind the reported errors:

• commoncrawl: getURLs returned false on any HTTP error, which breaked out of the loop over all 5 yearly indexes. Common Crawl's index frontend intermittently returns 502/504, and since the indexes were iterated from a map (random order), a single bad index would discard the remaining four at random. Now iterates years in order and keeps going past a failed index.
• waybackarchive: http.Client.Timeout was set from -timeout (default 30s), which bounds the entire exchange including the response body. Wayback's CDX API regularly needs longer than that just to send headers, and a full response streams for minutes, so it failed with context deadline exceeded ... while awaiting headers on essentially any real domain. -timeout now applies to connection setup (dial + TLS handshake). The request as a whole is still bounded by -max-time.

Also: both sources used bufio.Scanner at its default 64KB limit and never checked scanner.Err(), so an over-long CDX line silently truncated results. Added a larger buffer and error reporting. Tagged commoncrawl's error results with Type: source.Error. Without it enumerate.go treated them as URL results and never logged the warning.

Before / after on -d projectdiscovery.io, default flags:

# before                          # after
commoncrawl     0 results 1 err    commoncrawl     1414 results 0 err
commoncrawl     0 results 1 err    commoncrawl     1414 results 0 err
commoncrawl  ~1400 results 0 err   commoncrawl     1414 results 0 err
waybackarchive  0 results 1 err    waybackarchive 33360 results 0 err

Note wayback remains flaky upstream independent of this change. web.archive.org sometimes holds the connection ~4.5 min and then closes it without sending headers (EOF). That is server-side, so a retry could be a follow-up.

---

CI fixes (unrelated to #133, needed to get this PR green)

lint-test was already broken on dev. golangci-lint v1.64.8 is built with Go 1.24 and refuses a target above that, so it exited during config load:

can't load config: the Go language version (go1.24) used to build golangci-lint is lower than the targeted Go version (1.25.0)

Dependabot raised go.mod to go 1.25.0 in e91a803, but lint-test.yml skips bot actors, so every dependabot PR since reported skipping and nothing caught it. This is the first non-bot PR since that bump.

• Switched to projectdiscovery/actions/golangci-lint/v2@v1 (golangci-lint v2, golangci-lint-action@v9). v1.64.8 is the final v1 release, so there was no newer v1 to pin.
• That surfaced one pre-existing finding, govet inline on maps.Copy in pkg/agent/agent.go, caused by the //go:fix inline directives in golang.org/x/exp/maps. Migrated agent.go and stats.go to the stdlib maps package, which drops the deprecated dependency (go mod tidy demotes x/exp to indirect). Side benefit: the "Selected source(s)" debug line and the stats table are now deterministically ordered.

Summary by CodeRabbit

• Bug Fixes
  • Clarified that the timeout setting applies to establishing connections with sources.
  • Improved long-lived response streaming by preventing client-wide timeouts from interrupting active transfers.
  • Improved Common Crawl processing so one failed index does not stop other years from being searched.
  • Added clearer error reporting when archive responses cannot be scanned or processed.
  • Improved handling of large Common Crawl and Wayback Archive records.
  • Made archive index processing more consistent and reliable across searches.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c4dd1bcb-a14e-4236-b842-6e33a056c79e

📥 Commits

Reviewing files that changed from the base of the PR and between da5f632 and f6fa185.

⛔ Files ignored due to path filters (1)

• .github/workflows/lint-test.yml is excluded by !**/*.yml

📒 Files selected for processing (3)

• go.mod
• internal/runner/stats.go
• pkg/agent/agent.go

Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

---

Walkthrough

The change limits source connection setup, removes the client-wide HTTP timeout, expands CDX scanner buffers, reports scanner failures, continues Common Crawl index processing after individual errors, and replaces experimental map utilities.

Changes

Timeout and source processing

Layer / File(s)	Summary
Connection timeout behavior internal/runner/options.go, pkg/session/session.go	Timeout documentation now describes source connection time. The transport applies the timeout to dialing and TLS handshakes. Request contexts control request duration.
CDX scanning and error reporting pkg/source/commoncrawl/commoncrawl.go, pkg/source/waybackarchive/waybackarchive.go	Both sources accept larger CDX lines and report scanner errors as source.Error. Common Crawl processes indexes in year order and continues after individual index failures.
Standard-library map utilities go.mod, internal/runner/stats.go, pkg/agent/agent.go	Map utilities now use the standard library. Source names are sorted before debug logging and statistics rendering.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to f6fa1

The PR improves source error handling, timeout behavior, and CI compatibility without any actionable merge-blocking risk remaining after normal checks and review.

Suggested reviewers: mzack9999

Poem

A rabbit checks each source line,
With wider buffers set in place.
Connections keep their proper time,
While streams continue at their pace.
Ordered crawls report each flaw,
And maps now sort by standard law.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The source reliability changes are in scope. However, replacing experimental map usage in runner statistics and agent construction, plus moving golang.org/x/exp to indirect dependencies, is unrelated …	Remove the unrelated map and dependency cleanup, or provide a linked requirement that justifies these changes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 6 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The source changes directly address issue #133 by improving Common Crawl and Wayback Archive error handling, scanner limits, request behavior, and continuation after failures.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary change: fixing Common Crawl and Wayback Archive source errors.

Full details: Out of Scope Changes check

Explanation

The source reliability changes are in scope. However, replacing experimental map usage in runner statistics and agent construction, plus moving golang.org/x/exp to indirect dependencies, is unrelated cleanup not required by issue #133.

Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 6 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-commoncrawl-wayback-failures

---

_{Comment @coderabbitai help to get the list of available commands.}