encoder: support semi-planar 10-bit input on the CPU staging path - #240
Draft
crmurillo wants to merge 6 commits into
Draft
encoder: support semi-planar 10-bit input on the CPU staging path#240crmurillo wants to merge 6 commits into
crmurillo wants to merge 6 commits into
Conversation
The framework hardcoded --inputNumPlanes 3, which duplicated the flag when a sample passes its own value via extra_args. Only add the default when the sample does not override it. Add an H.265 Main10 test using a two-plane P010 input.
The 2 or 3 planes check printed a warning but continued execution. Encoding finished but generated garbage.
For 2-plane inputs the chroma plane holds interleaved Cb and Cr, so each chroma pixel spans two samples. Double the expected plane stride and the per-line byte count when copying the chroma plane.
The linear staging images are created outside of the video-format negotiation, so probe vkGetPhysicalDeviceImageFormatProperties2 before handing the format/tiling/usage combination to vkCreateImage. If the raw input format cannot back a linear staging image, disable the preprocess compute filter and fall back to CPU conversion, fail with VK_ERROR_FORMAT_NOT_SUPPORTED when even that format is unsupported. Restrict the staging image usage flags to what each path actually needs, and use separate format counts for the DPB, input and QP-map format queries.
Without the compute filter, LoadNextFrame could only convert 3-plane sources into a 2-plane encode format. When the input format matches the negotiated encode format, stage it with a direct plane copy instead, applying msbShift on the CPU during the copy since no GPU filter will do it afterwards. Reject a 2-plane input whose format does not match the encode format with a clear error instead of falling through to the planar converters.
For both (DG2 and GT1): - encode_h265_main10_profile and the new test encode_h265_main10_profile_p010 pass. - encode_av1_encode_feedback, encode_av1_encode_feedback_per_partition and decode_av1_argon_test9354_2 are not_supported. GT1: - encode_av1_main_profile_10bit pass. - encode_av1_high_profile and encode_av1_professional_profile properly marked as not_supported.
crmurillo
marked this pull request as draft
September 11, 2026 09:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The encoder previously only handled 3-plane sources on the CPU staging path, miscomputed the chroma stride for 2-plane input, and let an invalid --inputNumPlanes value fall through and produce garbage output. The linear staging images are now validated against the device's format support before creation, with a fallback to CPU conversion when the compute filter cannot be used. A P010 Main10 encode test is added and the Intel DG2 and GT1 results are updated.
Type of change
bug fix
Issue (optional)
Fixes #237
Tests
NVIDIA GeForce RTX 4060 Ti (10de:2803) / NVIDIA 595.44.00 / Ubuntu 24.04.4 LTS
Total Tests: 86
Passed: 74 (1 warning(s))
Crashed: 0
Failed: 0
Not Supported: 11
Skipped: 1 (in skip list)
Success Rate: 100.0%
AMD Radeon RX 7600 (RADV NAVI33) (1002:7480) / radv Mesa 26.3.0-devel (git-fe1491b758) / Ubuntu 24.04.4 LTS
Total Tests: 86
Passed: 70
Crashed: 0
Failed: 0
Not Supported: 12
Skipped: 4 (in skip list)
Success Rate: 100.0%
Intel(R) UHD Graphics 770 (ADL-S GT1) (8086:4680) / Intel open-source Mesa driver Mesa 26.3.0-devel (git-fe1491b758) / Ubuntu 24.04.4 LTS
Total Tests: 86
Passed: 59
Crashed: 0
Failed: 0
Not Supported: 25
Skipped: 2 (in skip list)
Success Rate: 100.0%
Intel(R) Arc(tm) A750 Graphics (DG2) (8086:56a1) / Intel open-source Mesa driver Mesa 26.3.0-devel (git-fe1491b758) / Ubuntu 24.04.4 LTS
Total Tests: 86
Passed: 63
Crashed: 0
Failed: 0
Not Supported: 21
Skipped: 2 (in skip list)
Success Rate: 100.0%