[krishung5]

Description

Add video inputs to the OpenEngine gRPC Generate path.

• Advertise video only when the model and runtime decoder support it.
• Map URL, data-URI, and raw-byte video sources while preserving global media order and caller UUIDs.
• Bound decoded video memory, including frame and embedded-audio storage.
• Add unit and one-GPU conformance coverage.

Stacked on #20446. Review the video-specific commits after the image branch. Audio remains out of scope.

Test Coverage

• 84 focused OpenEngine tests passed in the TensorRT-LLM release container.
• 73 media/input tests passed.
• Real one-GPU video validation passed before the final main refresh.
• Post-refresh model-backed conformance could not rerun locally because LLM_MODELS_ROOT is unavailable.

Related Issues

• Relates to DIS-2971

PR Checklist

• Reviewed the checklist; the change is backward compatible and includes regression coverage.

Dev Engineer Review

Adds OpenEngine image and video support with capability discovery, ordered media mapping, UUID preservation, decoded-memory limits, cancellation cleanup, and runtime decoder checks.

It also adds DNS validation for remote media and maps transient fetch failures to UNAVAILABLE. Main risks are asynchronous loading, cancellation, memory accounting, and runtime capability detection. Review severity counts are unavailable.

QA Engineer Review

Adds CPU unit coverage for multimodal mapping, decoding, ordering, UUIDs, limits, cancellation, errors, URL validation, and model-type resolution. Adds GPU conformance coverage for image and video generation with Qwen3-VL-2B.

test_multimodal.py is listed in l0_cpu.yml. test_multimodal_conformance.py is listed in l0_a10.yml. Reported validation includes 84 OpenEngine tests, 73 media/input tests, and one-GPU validation before the final main refresh. Post-refresh model-backed validation needs follow-up because LLM_MODELS_ROOT was unavailable. Coverage verdict: needs follow-up.

Per-File QA Perspective

• tensorrt_llm/grpc/openengine/capabilities.py: Verify modality discovery, placeholder formatting, and malformed-template failures.
• tensorrt_llm/grpc/openengine/control.py: Verify multimodal model-info reporting and sorted modalities.
• tensorrt_llm/grpc/openengine/errors.py: Verify MediaUnavailableError status mapping.
• tensorrt_llm/grpc/openengine/request_mapping.py: Verify media loading, ordering, UUIDs, limits, cancellation, decoder checks, and memory accounting.
• tensorrt_llm/grpc/openengine/servicer.py: Verify capability detection, request-ID reservation, cancellation handling, and UNAVAILABLE mapping.
• tensorrt_llm/inputs/media_io.py: Verify byte decoding, safe fetching, DNS validation, and decoded-frame and embedded-audio limits.
• tensorrt_llm/inputs/model_config.py: Verify class-level model_type precedence and fallback behavior.
• tensorrt_llm/inputs/utils.py: Verify media loaders use the shared safe-fetch path.
• tensorrt_llm/serve/chat_utils.py: Verify model-type resolution after helper relocation.
• tests/integration/test_lists/test-db/l0_a10.yml: Adds GPU multimodal conformance coverage and updates the model description.
• tests/integration/test_lists/test-db/l0_cpu.yml: Adds CPU multimodal coverage.
• tests/unittest/grpc/openengine/test_control.py: Covers default and image-enabled capability reporting.
• tests/unittest/grpc/openengine/test_multimodal.py: Covers CPU multimodal behavior, limits, failures, and regressions; listed in l0_cpu.yml.
• tests/unittest/grpc/openengine/test_multimodal_conformance.py: Covers real-engine image and video generation; listed in l0_a10.yml.
• tests/unittest/inputs/test_async_media_loading.py: Covers raw-byte image decoding, resolver behavior, and video memory-limit forwarding; no new test-list entry is identified.
• tests/unittest/inputs/test_model_config.py: Covers model-type precedence and fallback; no new test-list entry is identified.
• tests/unittest/inputs/test_url_validation.py: Covers transient fetch errors, resolver validation, private-address rejection, and session reuse; no new test-list entry is identified.
• tests/unittest/inputs/test_video_decode.py: Covers decoded-memory enforcement and buffer ownership; no new test-list entry is identified.

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80655 [ run ] triggered by Bot. Commit: 2d69a82 Link to invocation

[tensorrt-cicd]

PR_Github #80655 [ run ] completed with state SUCCESS. Commit: 2d69a82
/LLM/main/L0_MergeRequest_PR pipeline #66409 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80700 [ run ] triggered by Bot. Commit: 2d69a82 Link to invocation

[tensorrt-cicd]

PR_Github #80700 [ run ] completed with state SUCCESS. Commit: 2d69a82
/LLM/main/L0_MergeRequest_PR pipeline #66448 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80947 [ run ] triggered by Bot. Commit: 010abd4 Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80985 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

[tensorrt-cicd]

PR_Github #80947 [ run ] completed with state ABORTED. Commit: 010abd4

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80990 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

[tensorrt-cicd]

PR_Github #80985 [ run ] completed with state ABORTED. Commit: 38ad20c

Link to invocation

[tensorrt-cicd]

PR_Github #80990 [ run ] completed with state FAILURE. Commit: 38ad20c
/LLM/main/L0_MergeRequest_PR pipeline #66712 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81019 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

[tensorrt-cicd]

PR_Github #81019 [ run ] completed with state SUCCESS. Commit: 38ad20c
/LLM/main/L0_MergeRequest_PR pipeline #66736 completed with status: 'SUCCESS'

CI Report

Link to invocation