ffmpeg use kmsgrab on wayland not working

[chenjie199234]

i can't use ffmpeg's kmsgrab to capture screen on my pc
OS: Archlinux
WM: Sway 1.4
ffmpeg: n4.2.2
cpu: i5 7200u
gpu: the cpu's HD620
driver: intel-media-driver 19.4.0

here is my ffmpeg cmd:
exec on root:
ffmpeg -f kmsgrab -i - -vf 'hwdownload,format=bgr0' output.mp4

here is the out put:
ffmpeg version n4.2.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 9.2.1 (Arch Linux 9.2.1+20200130-2) 20200130
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3

libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100

[kmsgrab @ 0x5644e2b79fc0] Using plane 31 to locate framebuffers.
[kmsgrab @ 0x5644e2b79fc0] Failed to get framebuffer 122: Invalid argument.
pipe:: Invalid argument

i report this on sway's github:
​https://github.com/swaywm/sway/issues/4961

they said this is a ffmpeg's bug.

[Carl Eugen Hoyos]

Please test current FFmpeg git head to make this a valid ticket.

[chenjie199234]

the version is n4.2.2
it is released on github on 1 Jan 2020
take a look at the github release page

[Carl Eugen Hoyos]

Please reopen this ticket if the issue is reproducible with current FFmpeg git head, the only version supported here.

[Maxim Baz]

I can confirm that this issue is still reproducible on the latest master. I would be very happy to assist with getting to the bottom of it!

Relevant system info:

ffmpeg: latest master, 96e5e6abb9

Arch Linux
sway 1.4
Intel i7-1065G7 / Iris Plus Graphics (ICL GT2)
intel-media-driver 20.1.1
export LIBVA_DRIVER_NAME=iHD

Command with verbose logging enabled:

$ sudo ffmpeg -v 9 -loglevel 99 -f kmsgrab -i - -vf 'hwdownload,format=bgr0' output.mp4

ffmpeg version N-97647-g96e5e6abb9 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.0 (Arch Linux 9.3.0-1)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3 --enable-vulkan
  libavutil      56. 43.100 / 56. 43.100
  libavcodec     58. 82.100 / 58. 82.100
  libavformat    58. 43.100 / 58. 43.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 80.100 /  7. 80.100
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'kmsgrab'.
Reading option '-i' ... matched as input url with argument '-'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'hwdownload,format=bgr0'.
Reading option 'output.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input url -.
Applying option f (force format) with argument kmsgrab.
Successfully parsed a group of options.
Opening an input file: -.
[AVHWDeviceContext @ 0x5572fd811c00] Opened DRM device /dev/dri/card0: driver i915 version 1.6.0.
[kmsgrab @ 0x5572fd811140] Plane 31: CRTC 91 FB 256.
[kmsgrab @ 0x5572fd811140] Using plane 31 to locate framebuffers.
[kmsgrab @ 0x5572fd811140] Failed to get framebuffer 256: Invalid argument.
pipe:: Invalid argument

If that matters, if I run the command several times, the error will sometimes say "framebuffer 252" and sometimes "framebuffer 256".

Let me know how I can help!

[Maxim Baz]

To add some (hopefully helpful) details:

• sway maintainers mentioned that the root cause of the issue is that ffmpeg doesnt support modifiers [1]
• just like discussed here [2], Intel GPUs use I915_FORMAT_MOD_Y_TILED_CCS modifier (0x100000000000004), which uses two planes: compressed texture + color surface
• I confirm that my GPU uses exactly the same I915_FORMAT_MOD_Y_TILED_CCS modifier
• I saw that kmsgrab supports -format_modifier flag, but whether I use -format_modifier 0x100000000000004 or -format_modifier 72057594037927940 (decimal value) I still get the same error.
  • ffmpeg contains this in the verbose log: Reading option '-format_modifier' ... matched as AVOption 'format_modifier' with argument '0x100000000000004'.

Could it be just a misconfiguration on our side? Or a missing support for this specific DRM modifier?

1: ​https://github.com/swaywm/sway/issues/4961#issuecomment-579903298
2: ​https://github.com/swaywm/wlroots/issues/1965#issuecomment-597619178

[Lynne]

The issue is Sway decides to use the new compressed plane modifiers which Intel uses for power savings, however, kmsgrab doesn't yet support capturing those, but even if it did, you wouldn't be able to do anything with them since neither Vulkan, nor VAAPI, nor OpenCL supports such surfaces yet. Vulkan seems to be the closest to supporting those, all we need is for Mesa to implement support for that. VAAPI doesn't seem to have a patch to support those yet.
You can patch your sway to avoid using those modifiers for some unnoticeable lossless in power savings.
So, its not our bug for the moment. Once some API supports importing such DMABUFs we can change kmsgrab.

[Maxim Baz]

Thank you for the reply! This is very insightful, because I have actually been playing with a side-project lately and as part of it was trying to import DMA-BUFs as an image on my hardware.

I know it is possible to import DMA-BUFs using EGL, for example see wlr_egl_create_image_from_dmabuf() function in wlroots [1] (I confirmed that it works as expected).

I have spent the last two days learning Vulkan and trying to import DMA-BUFs using it instead of EGL, and have actually been using libavutil/hwcontext_vulkan.c from ffmpeg as a reference ( vulkan_map_from_drm_frame_desc() function) - that's how I stumbled upon this bug, because I wanted to confirm that the code in ffmpeg itself is able to import the image :)

From what I've understood so far, vulkaninfo tells me that my GPU supports VK_EXT_external_memory_dma_buf but doesn't support VK_EXT_image_drm_format_modifier . I have already had some suspicions that vulkan_map_from_drm_frame_desc() won't be able to import DMA-BUFs on my hardware, because it creates multi-planar VkImageCreateInfo structure only when VK_EXT_image_drm_format_modifier bit is supported (see line 1764 currently), which definitely isn't the case with my GPU.

Do you think it is something that can be fixed, or importing such multi-planar DMA-BUFs is fundamentally impossible using Vulkan API nowadays? I would appreciate some pointers, as I am very new to these topics, but I'm eager to get to the bottom of it. I would be happy to experiment with importing DMA-BUFs using Vulkan, and contribute back to your code in libavutil/hwcontext_vulkan.c , even if it won't fully fix this ticket, it will hopefully be a step in the right direction.

Patching sway as you suggested is of course an option, but if I can make it work without patching, I would of course prefer that.

1: ​https://github.com/swaywm/wlroots/blob/6357e166f9a31f0c828ea5bb0000627c15442505/render/egl.c#L477

[Lynne]

The Mesa PR is here: ​https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1466
It might say its 2 months old but it is in fact 2 years old, and no matter how much I've asked for it to be merged on IRC its development is beyond sluggish.
Its a really big blocker to doing anything useful with Vulkan on linux. I too, really need this merged to get my project working.
The Vulkan API with the modifiers EXT is fully capable of dealing with any DMABUF thrown at it. And the hwcontext can too, deal with any DMABUF thrown at it. Its pedantically correct according to the specs. It definitely does not produce multiplane images however, merely a single image composed of multiple objects (dmabuf fd mapped as VulkanMemory).
In case of multiplane images (e.g. NV12 or yuv420p), it produces one image per plane, so still not multiplanar. Multiplanar images in Vulkan are generally crap and you shouldn't use them, even according to official Intel devs.
Unfortunately, without modifiers, Vulkan assumes all images on Intel use either Y tiling or linear images. And the only thing producing Y tiled images on Intel is VAAPI. Hence why only VAAPI->DMABUF->Vulkan works. Compositors _cannot_ draw to Y tiled buffers, only to X tiled buffers. Even if you disable compressed surfaces, and you try to import a display DMABUF, you'll get garbage out, as the tiling will be wrong. VAAPI will quite happily work though.

In the meantime, I'm writing this: ​https://github.com/cyanreg/txproto. It uses the sway export dmabuf extension and the screencopy extension to capture the screen and filter/encode it, although it can use the kmsgrab libavdevice library too. If you don't want to modify wlroots to disable the modifiers (I can give a patch on IRC), you can use the screencapture protocol to upload the image to either Vulkan, OpenCL or VAAPI (or even CUDA) and encode it that way, along with sound (and unlike ffmpeg, it'll have frame-perfect audio-video sync).