fix(download): classify relative media URLs instead of throwing - #2475
Open
MaxFreedomPollard wants to merge 1 commit into
Open
fix(download): classify relative media URLs instead of throwing#2475MaxFreedomPollard wants to merge 1 commit into
MaxFreedomPollard wants to merge 1 commit into
Conversation
detectContentType() built its extension from `new URL(url).pathname` (src/download/index.ts:92), which throws TypeError: Invalid URL for a relative path like /media/photo.jpg or a protocol-relative one like //cdn.example.com/clip.mp4. Adapters produce those whenever a site's JSON returns unqualified media paths. The pipeline download step calls detectContentType() at src/pipeline/steps/download.ts:238, outside the per-item try block, so the TypeError escaped mapConcurrent and rejected the whole step: one bad row killed every other download in the batch, and the step's own per-item "failed" result was never produced. requiresYtdlp() next to it already tolerates unparseable URLs; detectContentType() did not. Extension sniffing now goes through urlPathForExtension(), which falls back to trimming the query and hash by hand when new URL() rejects the input, so relative media URLs classify by extension and a genuinely unusable URL fails as one item instead of as the batch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
detectContentType()builds the extension it classifies on fromnew URL(url).pathname(src/download/index.ts:92).new URL()rejects a relative path like/media/photo.jpgand a protocol-relative one like//cdn.example.com/clip.mp4, so the call throwsTypeError: Invalid URLon exactly the values adapters produce when a site's JSON returns unqualified media paths.The pipeline download step calls it at
src/pipeline/steps/download.ts:238, which sits above the per-itemtryblock. TheTypeErrorescapes themapConcurrentworker and rejects the whole step, so one bad row kills every other download in the batch and the step's own{ status: 'failed' }result for that row is never produced.requiresYtdlp()sitting right next to it already tolerates unparseable URLs (there is a test for that);detectContentType()did not, even though the test above it calls both on the same URLs.Extension sniffing now goes through a small
urlPathForExtension()helper that falls back to trimming the query and hash by hand whennew URL()rejects the input./media/photo.jpgclassifies asimageand//cdn.example.com/clip.mp4asvideo, and a genuinely unusable URL now fails as one item (httpDownloadreports it) instead of as the batch.Related issue:
Type of Change
Checklist
Screenshots / Output
Three regression tests were added. On
mainwithout the source change all three fail withTypeError: Invalid URLraised atsrc/download/index.ts:92, the pipeline one throughmapConcurrentatsrc/utils.ts:31:With the fix: