Skip to content

fix(download): classify relative media URLs instead of throwing - #2475

Open
MaxFreedomPollard wants to merge 1 commit into
jackwener:mainfrom
MaxFreedomPollard:fix/download-content-type-invalid-url
Open

fix(download): classify relative media URLs instead of throwing#2475
MaxFreedomPollard wants to merge 1 commit into
jackwener:mainfrom
MaxFreedomPollard:fix/download-content-type-invalid-url

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown

Description

detectContentType() builds the extension it classifies on from new URL(url).pathname (src/download/index.ts:92). new URL() rejects a relative path like /media/photo.jpg and a protocol-relative one like //cdn.example.com/clip.mp4, so the call throws TypeError: Invalid URL on exactly the values adapters produce when a site's JSON returns unqualified media paths.

The pipeline download step calls it at src/pipeline/steps/download.ts:238, which sits above the per-item try block. The TypeError escapes the mapConcurrent worker and rejects the whole step, so one bad row kills every other download in the batch and the step's own { status: 'failed' } result for that row is never produced. requiresYtdlp() sitting right next to it already tolerates unparseable URLs (there is a test for that); detectContentType() did not, even though the test above it calls both on the same URLs.

Extension sniffing now goes through a small urlPathForExtension() helper that falls back to trimming the query and hash by hand when new URL() rejects the input. /media/photo.jpg classifies as image and //cdn.example.com/clip.mp4 as video, and a genuinely unusable URL now fails as one item (httpDownload reports it) instead of as the batch.

Related issue:

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 🌐 New site adapter
  • 📝 Documentation
  • ♻️ Refactor
  • 🔧 CI / build / tooling

Checklist

  • I ran the checks relevant to this PR
  • I updated tests or docs if needed
  • I included output or screenshots when useful

Screenshots / Output

Three regression tests were added. On main without the source change all three fail with TypeError: Invalid URL raised at src/download/index.ts:92, the pipeline one through mapConcurrent at src/utils.ts:31:

FAIL |unit| src/download/index.test.ts > returns false for an unparseable URL instead of throwing
FAIL |unit| src/download/index.test.ts > classifies relative and protocol-relative media URLs by extension
FAIL |unit| src/pipeline/steps/download.test.ts > keeps downloading the rest of a batch when one item has an unparseable URL
Tests  3 failed | 12 passed (15)

With the fix:

npx vitest run --project unit src/download/index.test.ts src/pipeline/steps/download.test.ts
Test Files  2 passed (2)
Tests  15 passed (15)

npx tsc --noEmit
(clean)

npm test
Test Files  631 passed (631)
Tests  7318 passed | 1 skipped (7319)

npm run check:silent-column-drop
Silent-column-drop gate: current=94, baseline=94, new=0, resolved=0

npm run check:typed-error-lint
Typed-error lint gate: current=130, baseline=130, new=0, resolved=0

npm run build && git diff --exit-code -- cli-manifest.json
(no drift)

detectContentType() built its extension from `new URL(url).pathname`
(src/download/index.ts:92), which throws TypeError: Invalid URL for a
relative path like /media/photo.jpg or a protocol-relative one like
//cdn.example.com/clip.mp4. Adapters produce those whenever a site's JSON
returns unqualified media paths.

The pipeline download step calls detectContentType() at
src/pipeline/steps/download.ts:238, outside the per-item try block, so the
TypeError escaped mapConcurrent and rejected the whole step: one bad row
killed every other download in the batch, and the step's own per-item
"failed" result was never produced. requiresYtdlp() next to it already
tolerates unparseable URLs; detectContentType() did not.

Extension sniffing now goes through urlPathForExtension(), which falls back
to trimming the query and hash by hand when new URL() rejects the input, so
relative media URLs classify by extension and a genuinely unusable URL fails
as one item instead of as the batch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant