VAAPI decoding on AMD GPU may cause cpu soft lockup

[sunboeasy@gmail.com]

Summary of the bug:
Our software uses FFmpeg for hardware decoding. One user encountered a CPU stuck issue while using it and found that the same problem occurs when using FFmpeg command line directly.
Here is the link of the related issue: ​https://github.com/rustdesk/rustdesk/issues/6906#issuecomment-1897921793

How to reproduce:

% ffmpeg -hwaccel vaapi  -i 1920_1080.264 -vf 'format=nv12' -c:v rawvideo -pix_fmt nv12 output.yuv
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)

Logs:

DRI_PRIME=1 ffmpeg -hwaccel vaapi  -i 1920_1080.264 -vf 'format=nv12' -c:v rawvideo -pix_fmt nv12 output.yuv
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[h264 @ 0x563618de36c0] decoding for stream 0 failed
Input #0, h264, from '1920_1080.264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 60 tbr, 1200k tbn, 120 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
radeon: The kernel rejected CS, see dmesg for more information (-22).
radeon: The kernel rejected CS, see dmesg for more information (-2).
Output #0, rawvideo, to 'output.yuv':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 1492992 kb/s, 60 fps, 60 tbn
    Metadata:
      encoder         : Lavc58.134.100 rawvideo
frame=    1 fps=0.0 q=-0.0 Lsize=    3038kB time=00:00:00.01 bitrate=1492962.1kbits/s speed=0.291x    
video:3038kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

----> CPU Lockup here

Please refer to the provided link for more information, including 'vainfo' and other details

[elenril]

radeon: The kernel rejected CS, see dmesg for more information (-22).

This sounds like a kernel or hardware issue, there is nothing we can do about it.