kmsgrab pixel formats missing DRM_FORMAT_ARGB2101010 and others

[jean]

Summary of the bug: When a DRM plane is using certain formats found in modern wayland compositors, kmsgrab will fail to recognize the pixel format and refuse to start.
How to reproduce:
On KDE Plasma wayland 5.26.5:

% sudo ffmpeg -f kmsgrab -i - output.mp4
ffmpeg version N-109841-ge3bbf5c17d Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.1 (GCC) 20230201
...
[kmsgrab @ 0x559fd171b2c0] Framebuffer pixel format 30335241 is not a known supported format.

This was tested on Arch Linux.

The specific pixfmt missing in my case was DRM_FORMAT_ARGB2101010, though it's possible other formats of this classification (32 bpp format with 10 bits per color and 2 bits of alpha) might also be used and would also need to be added.

[bermond]

DRM format 30334241 is also missing.

The command bellow fails in Wayland and runs fine under X11:

$ ffmpeg -device /dev/dri/card1 -f kmsgrab -i '-' -vf 'hwmap=derive_device=vaapi,crop=1280:720:600:10,scale_vaapi=1280:720:nv12' -c:v h264_vaapi test-out.mp4
ffmpeg version N-114554-g7bf85d2d3a Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13.2.1 (GCC) 20230801
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --disable-htmlpages --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      59. 10.100 / 59. 10.100
  libavcodec     61.  5.100 / 61.  5.100
  libavformat    61.  2.100 / 61.  2.100
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.100 / 10.  2.100
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
  libpostproc    58.  2.100 / 58.  2.100
[kmsgrab @ 0x5f3360f73800] Using plane 31 to locate framebuffers.
[kmsgrab @ 0x5f3360f73800] Template framebuffer is 261: 1920x1080 format 30334241 modifier 100000000000002 flags 2.
[kmsgrab @ 0x5f3360f73800] Framebuffer pixel format 30334241 is not a known supported format.
[in#0 @ 0x5f3360f3a600] Error opening input: Invalid argument
Error opening input file -.
Error opening input files: Invalid argument

• OS: Arch Linux
• Desktop: Plasma 6.0.3
• Session: Wayland

[bermond]

It looks like that format 30334241 is DRM format DRM_FORMAT_ABGR2101010. Although missing the 0x prefix, the number is actually in hexadecimal, and not in decimal as it may suggest.

The following 32bpp-RGB DRM formats are missing:

• DRM_FORMAT_XBGR2101010
• DRM_FORMAT_RGBX1010102
• DRM_FORMAT_BGRX1010102
• DRM_FORMAT_ARGB2101010
• DRM_FORMAT_ABGR2101010
• DRM_FORMAT_RGBA1010102
• DRM_FORMAT_BGRA1010102

Among these formats, DRM_FORMAT_ARGB2101010 and DRM_FORMAT_ABGR2101010 seems to be more common, and they are also the ones informed here in the user commands above. A ​patch was proposed in 2019 for adding them, but it was rejected.

But it looks like the solution is not so simple as just adding these formats. VAAPI/libva/Mesa also needs to support them, and it appears the support is equally missing there. An ​issue is opened in the libva repository requesting them since 2019, without any answer from upstream. There seems to be some complexity involved, and the problem is also discussed in ticket #8542, being described there as ​not being a ffmpeg bug in the current situation.