Skip to content

[BUG] Pixelflux 1.6.4 VAAPI initialization fails on Intel UHD 630 with VAProfile is not supported #170

Description

@cerede2000

Is there an existing issue for this?

  • I have searched the existing issues

I found several Intel VAAPI-related issues, but none matching this exact failure:

Failed to create processing pipeline config: 12
(the requested VAProfile is not supported)

Current Behavior

When using PixelFlux Wayland with H.264 VAAPI encoding on an Intel UHD Graphics 630, the VAAPI encoder initialization systematically fails.

Selkies then falls back to CPU H.264 encoding:

[Wayland] Initializing Unified VAAPI Encoder...
[Parsed_scale_vaapi_1 @ 0x75335475fdc0] Failed to create processing pipeline config: 12 (the requested VAProfile is not supported).
[Parsed_scale_vaapi_1 @ 0x75335475fdc0] Failed to configure output pad on Parsed_scale_vaapi_1
[Wayland] Failed to init VAAPI: Failed to config filter graph. Falling back to CPU.
[Wayland] Decision: No GPU Encoder available -> Using CPU Software Encoding.
Stream settings active -> Res: 3008x1730 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver CRF: 18 (Burst: 5f) | Colorspace: I420 (Limited Range)

The same error occurs with multiple resolutions, including standard and aligned resolutions:

1280x720
1920x1080
2184x1728
2184x1730
3008x1728
3008x1730

The issue is therefore not dependent on the output resolution.

Expected Behavior

PixelFlux should initialize the Intel VAAPI H.264 encoder successfully and report something equivalent to:

VAAPI Encoder Initialized successfully.
Mode: H264 (VAAPI) FullFrame

The Intel GPU should be used for H.264 stream encoding instead of falling back to CPU encoding.

Steps To Reproduce

  1. Run the latest LinuxServer Microsoft Edge image with the Intel GPU passed through:
services:
  msedge:
    image: lscr.io/linuxserver/msedge:latest
    container_name: msedge

    devices:
      - /dev/dri:/dev/dri

    environment:
      PIXELFLUX_WAYLAND: "true"
      AUTO_GPU: "false"

      DRINODE: "/dev/dri/renderD128"
      DRI_NODE: "/dev/dri/renderD128"
      LIBVA_DRIVER_NAME: "iHD"

      SELKIES_ENCODER: "x264enc"
      SELKIES_USE_CPU: "false"
      SELKIES_H264_FULLCOLOR: "false"
      SELKIES_H264_STREAMING_MODE: "false"
      SELKIES_FORCE_ALIGNED_RESOLUTION: "true"
  1. Start or recreate the container:
docker compose up -d --force-recreate msedge
  1. Connect to the Selkies web interface.

  2. Select H.264 encoding with CPU encoding disabled.

  3. Check the container logs:

docker logs -f msedge 2>&1 |
grep --line-buffered -Ei \
'VAAPI|scale_vaapi|GPU Encoder|CPU Software|Stream settings'
  1. VAAPI initialization fails and Selkies falls back to H.264 CPU encoding.

Environment

Container image


lscr.io/linuxserver/msedge:latest


### Container operating system


PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
VERSION_ID="13"
DEBIAN_VERSION_FULL=13.5


### Selkies components


pixelflux=1.6.4
selkies=0.0.0
pcmflux=1.0.8


### CPU architecture


x86-64


### GPU


Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e91]


### Kernel driver


Kernel driver in use: i915


### DRM devices inside the container


crw-rw---- root video 226,   0 /dev/dri/card0
crw-rw---- root sgx   226, 128 /dev/dri/renderD128


The LinuxServer initialization correctly adds the container user to the relevant groups:


adding /dev/dri/renderD128 to video group with id 993
adding /dev/dri/card0 to video group video with id 44


## Effective environment variables

The variables are correctly present inside the running container:


AUTO_GPU=false
DRINODE=/dev/dri/renderD128
DRI_NODE=/dev/dri/renderD128
LIBVA_DRIVER_NAME=iHD
PIXELFLUX_WAYLAND=true
SELKIES_ENCODER=x264enc
SELKIES_FORCE_ALIGNED_RESOLUTION=true
SELKIES_H264_FULLCOLOR=false
SELKIES_H264_STREAMING_MODE=false
SELKIES_USE_CPU=false


Selkies also parses the settings correctly:


encoder: x264enc
force_aligned_resolution: True
h264_fullcolor: False
h264_streaming_mode: False
use_cpu: False
dri_node: /dev/dri/renderD128


## Independent VAAPI validation

The host, Docker device mapping, Intel VAAPI driver, `scale_vaapi`, and `h264_vaapi` have all been validated successfully using the LinuxServer FFmpeg image.

### Successful `iHD` VAAPI test


docker run --rm -it \
  --device=/dev/dri:/dev/dri \
  -e LIBVA_DRIVER_NAME=iHD \
  lscr.io/linuxserver/ffmpeg:latest \
  -hide_banner \
  -loglevel verbose \
  -vaapi_device /dev/dri/renderD128 \
  -f lavfi \
  -i testsrc2=size=1920x1080:rate=60 \
  -vf 'format=nv12,hwupload,scale_vaapi=w=1280:h=720:format=nv12' \
  -t 10 \
  -c:v h264_vaapi \
  -f null -


Relevant successful output:


libva: VA-API version 1.23.0
libva: User environment variable requested driver 'iHD'
libva: va_openDriver() returns 0
Initialised VAAPI connection: version 1.23
VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 26.1.5

Using VAAPI profile VAProfileH264High
Using VAAPI entrypoint VAEntrypointEncSlice
Using VAAPI render target format YUV420

Output:
Video: h264 (High), vaapi(tv, progressive), 1280x720, 60 fps

600 frames encoded
Exit code 0


Performance:


frame=600
fps=333
speed=5.55x


### Successful `scale_vaapi` passthrough test

The same test also succeeds without resizing:


-vf 'format=nv12,hwupload,scale_vaapi=w=1920:h=1080:format=nv12'


Relevant output:


scale_vaapi: Using VAAPI filter passthrough mode.
Video: h264 (High), vaapi(tv, progressive), 1920x1080, 60 fps
600 frames encoded
Exit code 0


### Successful `i965` test

The legacy Intel driver was also tested successfully:


VAAPI driver: Intel i965 driver for Intel(R) Coffee Lake - 2.4.1
Using VAAPI profile VAProfileH264High
Using VAAPI entrypoint VAEntrypointEncSlice
600 frames encoded
Exit code 0


However, when `LIBVA_DRIVER_NAME=i965` is used in the actual `msedge` container, PixelFlux fails earlier:


Failed to initialise VAAPI connection: -1 (unknown libva error).
Failed to derive VAAPI device: Input/output error.


For that reason, `iHD` is used for the main reproduction case.

## Additional observations

The issue occurs with:


SELKIES_H264_FULLCOLOR=false
SELKIES_USE_CPU=false
SELKIES_FORCE_ALIGNED_RESOLUTION=true


The resulting stream is correctly requested as:


Colorspace: I420 (Limited Range)


This does not appear to be related to H.264 4:4:4 or full-color mode.

The failure occurs specifically during PixelFlux's internal VAAPI filter graph initialization:


Parsed_scale_vaapi_1
Failed to create processing pipeline config


A standalone FFmpeg pipeline using:


format=nv12,hwupload,scale_vaapi


works correctly on the same GPU, host, Docker runtime, render node, and Intel driver.

This suggests that PixelFlux may be creating a VAAPI frames context, surface format, processing profile, or filter graph configuration that differs from the working standalone FFmpeg pipeline.

Docker creation

services:
  msedge:
    image: lscr.io/linuxserver/msedge:latest
    container_name: msedge
    restart: unless-stopped

    devices:
      - /dev/dri:/dev/dri

    environment:
      PUID: "911"
      PGID: "911"
      TZ: "Europe/Paris"

      PIXELFLUX_WAYLAND: "true"
      AUTO_GPU: "false"

      DRINODE: "/dev/dri/renderD128"
      DRI_NODE: "/dev/dri/renderD128"
      LIBVA_DRIVER_NAME: "iHD"

      SELKIES_ENCODER: "x264enc"
      SELKIES_USE_CPU: "false"
      SELKIES_H264_FULLCOLOR: "false"
      SELKIES_H264_STREAMING_MODE: "false"
      SELKIES_FORCE_ALIGNED_RESOLUTION: "true"

    volumes:
      - /path/to/msedge/config:/config

    ports:
      - "3000:3000"
      - "3001:3001"


Container recreated with:


docker compose pull msedge

docker compose up -d \
  --no-deps \
  --force-recreate \
  msedge


The GPU devices are visible inside the container:


/dev/dri/card0
/dev/dri/renderD128

Container logs

INFO:main:Initial Encoder: x264enc, Framerate: 60

[Wayland] Initializing Unified VAAPI Encoder...
[Parsed_scale_vaapi_1 @ 0x75335475fdc0] Failed to create processing pipeline config: 12 (the requested VAProfile is not supported).
[Parsed_scale_vaapi_1 @ 0x75335475fdc0] Failed to configure output pad on Parsed_scale_vaapi_1
[Wayland] Failed to init VAAPI: Failed to config filter graph. Falling back to CPU.
[Wayland] Decision: No GPU Encoder available -> Using CPU Software Encoding.
Stream settings active -> Res: 3008x1730 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver CRF: 18 (Burst: 5f) | Colorspace: I420 (Limited Range)

[Wayland] Initializing Unified VAAPI Encoder...
[Parsed_scale_vaapi_1 @ 0x753355553000] Failed to create processing pipeline config: 12 (the requested VAProfile is not supported).
[Parsed_scale_vaapi_1 @ 0x753355553000] Failed to configure output pad on Parsed_scale_vaapi_1
[Wayland] Failed to init VAAPI: Failed to config filter graph. Falling back to CPU.
[Wayland] Decision: No GPU Encoder available -> Using CPU Software Encoding.
Stream settings active -> Res: 3008x1728 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver CRF: 18 (Burst: 5f) | Colorspace: I420 (Limited Range)

[Wayland] Initializing Unified VAAPI Encoder...
[Parsed_scale_vaapi_1 @ 0x753356164480] Failed to create processing pipeline config: 12 (the requested VAProfile is not supported).
[Parsed_scale_vaapi_1 @ 0x753356164480] Failed to configure output pad on Parsed_scale_vaapi_1
[Wayland] Failed to init VAAPI: Failed to config filter graph. Falling back to CPU.
[Wayland] Decision: No GPU Encoder available -> Using CPU Software Encoding.
Stream settings active -> Res: 1920x1080 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver CRF: 18 (Burst: 5f) | Colorspace: I420 (Limited Range)

[Wayland] Initializing Unified VAAPI Encoder...
[Parsed_scale_vaapi_1 @ 0x7532febb83c0] Failed to create processing pipeline config: 12 (the requested VAProfile is not supported).
[Parsed_scale_vaapi_1 @ 0x7532febb83c0] Failed to configure output pad on Parsed_scale_vaapi_1
[Wayland] Failed to init VAAPI: Failed to config filter graph. Falling back to CPU.
[Wayland] Decision: No GPU Encoder available -> Using CPU Software Encoding.
Stream settings active -> Res: 1280x720 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver CRF: 18 (Burst: 5f) | Colorspace: I420 (Limited Range)

Workaround

The current workaround is to explicitly force CPU H.264 encoding:

SELKIES_ENCODER: "x264enc"
SELKIES_USE_CPU: "true"

This avoids repeated VAAPI initialization failures, but causes significant CPU usage at high resolutions and 60 FPS.

Request

Could you confirm whether PixelFlux 1.6.4 creates a different VAAPI input surface or filter graph for Wayland zero-copy capture than the standard working FFmpeg pipeline?

It would be useful to expose the complete FFmpeg filter graph and VAAPI pixel formats selected by the Unified VAAPI Encoder when debug logging is enabled.

A fallback pipeline similar to the following appears to work correctly on this hardware:

format=nv12,hwupload,scale_vaapi=format=nv12,h264_vaapi

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions