Opened 7 years ago
Closed 5 years ago
#7030 closed defect (fixed)
qsv in ffmpeg
Reported by: | Palich | Owned by: | |
---|---|---|---|
Priority: | important | Component: | avcodec |
Version: | git-master | Keywords: | qsv regression |
Cc: | zhong.li@intel.com, linjie.fu@intel.com | Blocked By: | |
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Testing was conducted on the same equipment on the same video stream.
configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-testinclude -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-testlib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-libmp3lame --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --enable-runtime-cpudetect --enable-decklink --enable-libmfx
ffmpeg version 3.2.10
ffmpeg -i rtmp://...
-c:v h264_qsv -b:v 1000k -g 50 -r 25
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
%CPU LOAD (top) 29,7
%GPU LOAD (metrics_monitor)
RENDER usage: 15.15, VIDEO usage: 6.06, VIDEO_E usage: 0.00 GT Freq: 350.00
This works well.
ffmpeg version 3.4.2
add -init_hw_device qsv:hw
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
%CPU LOAD (top) 36.0
%GPU LOAD (metrics_monitor)
RENDER usage: 14.14, VIDEO usage: 5.05, VIDEO_E usage: 0.00 GT Freq: 350.00
This works well.
ffmpeg version 3.3.6
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
Unrecognized option 'init_hw_device'.
Without this option, QSV is not initialized.
And gives an error. Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
ffmpeg version N-90085-gacdea9e
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
How to make QSV work in these versions ? (3.3.6 and N-90085-gacdea9e)
./sys_analyzer_linux.py -------------------------- Hardware readiness checks: -------------------------- [ OK ] Processor name: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz -------------------------- OS readiness checks: -------------------------- [ OK ] GPU visible to OS -------------------------- Media Server Studio Install: -------------------------- [ OK ] user in video group [ OK ] libva.so.1 found [ OK ] vainfo reports valid codec entry points [ OK ] /dev/dri/renderD128 connects to Intel i915 -------------------------- Component Smoke Tests: -------------------------- [ OK ] Media SDK HW API level:1.19 [ OK ] Media SDK SW API level:1.19 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK
Attachments (1)
Change History (26)
comment:1 by , 7 years ago
Keywords: | qsv regression added |
---|---|
Version: | 3.4 → git-master |
comment:4 by , 7 years ago
git bisect good
b0cd14fb1dab4b044f7fe6b53ac635409849de77 is the first bad commit
commit b0cd14fb1dab4b044f7fe6b53ac635409849de77
Author: Mark Thompson <sw@jkqxz.net>
Date: Thu Oct 26 00:18:47 2017 +0100
ffmpeg: Use codec hardware config to configure hwaccels
Removes specific support for all hwaccels supported by the generic code
(DXVA2, D3D11VA, NVDEC, VAAPI and VDPAU).
:040000 040000 db8084af9c0723e94edafc6d4beb29386dee6288 a1b4cd2717b8f391249024ea335241ac7788f843 M fftools
comment:5 by , 7 years ago
Status: | new → open |
---|
comment:6 by , 7 years ago
The author of the commit Mark Thompson <sw@jkqxz.net> had some grandiose idea but he did not realize it to the end. I do not know how the hardware decoders work, but the hardware coders do not work after this commit. The function hw_device_setup_for_encode suggests that there is a hardware configuration for each encoder, but they are not declared. And hardware encoders do not connect to the codec. This should either be removed or completed.
comment:7 by , 7 years ago
This patch solves the problem, but only for h264_qsv encoder.
diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index 09e4c0e..5f60675 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -34,6 +34,7 @@ #include "qsv.h" #include "qsv_internal.h" #include "qsvenc.h" +#include "hwaccel.h" typedef struct QSVH264EncContext { AVClass *class; @@ -169,6 +170,16 @@ static const AVCodecDefault qsv_enc_defaults[] = { { NULL }, }; +#define HWACCEL_QSV(codec) \ + HW_CONFIG_HWACCEL(1, 1, 1, QSV, QSV, ff_ ## codec ## _qsv_hwaccel) + +const AVHWAccel ff_h264_qsv_hwaccel = { + .name = "h264_qsv", + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_H264, + .pix_fmt = AV_PIX_FMT_QSV +}; + AVCodec ff_h264_qsv_encoder = { .name = "h264_qsv", .long_name = NULL_IF_CONFIG_SMALL("H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)"), @@ -186,5 +197,9 @@ AVCodec ff_h264_qsv_encoder = { .priv_class = &class, .defaults = qsv_enc_defaults, .caps_internal = FF_CODEC_CAP_INIT_CLEANUP, + .hw_configs = (const AVCodecHWConfigInternal*[]) { + HWACCEL_QSV(h264), + NULL + }, .wrapper_name = "qsv", };
comment:8 by , 7 years ago
Please send your patch - made with git format-patch
- to the FFmpeg development mailing list where it can be reviewed.
comment:10 by , 7 years ago
This works fine for me. At a guess you are somehow opening the DRM master device rather than a render node and therefore the unused top-level device has exclusive access to the GPU and stops the encoder from opening it?
If that's the problem then not creating the unused top-level device (remove the -init_hw_device
option) should make it work for you. (That device did automatically get passed to an encoder in 3.4, but there was disagreement about that feature being useful because ffmpeg was unable to tell whether the device was required or not (it isn't here, the internal session works for isolated encoders) and offered unhelpful warnings.)
comment:11 by , 7 years ago
Without this parameter, the situation is as follows.
The codec tries to code programmatically and it does not work.
[h264_qsv @ 0x1e87ec0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode. Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height [aac @ 0x1e8a540] Qavg: 62368.684
comment:12 by , 7 years ago
Then I suggest providing more information about your setup, and a complete log of a failing command.
comment:14 by , 7 years ago
The message even has a hint of what to do - try choosing a different rate control mode.
comment:15 by , 7 years ago
Believe me, this will not help.
You can choose the parameters for which there will be no error. But the encoding will occur on the CPU and not on the GPU.
comment:16 by , 7 years ago
Replying to palich2000:
Look above, there I described everything.
If might help if you could increase the scope of "everything" to include:
- Operating system details, including kernel version.
- What version of the Media SDK you are using (some specific proprietary version, the open-source one?).
- A complete log of ffmpeg output for the failing case.
comment:17 by , 7 years ago
Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[ OK ] Media SDK HW API level:1.19 [ OK ] Media SDK SW API level:1.19 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK
ffmpeg -t 30 -v verbose -i rtmp://xxxxx \ -c:v h264_qsv -b:v 1000k\ -strict experimental -acodec aac -ab 64k -ar 44100 \ -f flv rtmp://127.0.0.1/vip/_77_
ffmpeg version N-90198-g7056d06 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11) configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-test//include -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-test//lib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --disable-doc --enable-libmp3lame --disable-stripping --enable-debug=3 --enable-runtime-cpudetect --enable-libmfx libavutil 56. 8.100 / 56. 8.100 libavcodec 58. 13.100 / 58. 13.100 libavformat 58. 10.100 / 58. 10.100 libavdevice 58. 2.100 / 58. 2.100 libavfilter 7. 12.100 / 7. 12.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 0.101 / 5. 0.101 libswresample 3. 0.101 / 3. 0.101 libpostproc 55. 0.100 / 55. 0.100 Routing option strict to both codec and muxer layer Parsing... Parsed protocol: 0 Parsed host : vr0.dpl-streaming.com Parsed app : virtual RTMP_Connect1, ... connected, handshaking HandShake: Type Answer : 03 HandShake: Server Uptime : 134225664 HandShake: FMS Version : 0.0.0.0 HandShake: Handshaking finished.... RTMP_Connect1, handshaked Invoking connect HandleServerBW: server BW = 5000000 HandleClientBW: client BW = 5000000 2 HandleChangeChunkSize, received: chunk size change to 4096 RTMP_ClientPacket, received: invoke 190 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 1.00> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: fmsVer, STRING: FMS/3,0,1,123> Property: <Name: capabilities, NUMBER: 31.00> (object end) Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetConnection.Connect.Success> Property: <Name: description, STRING: Connection succeeded.> Property: <Name: objectEncoding, NUMBER: 0.00> (object end) (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <connect> sending ctrl. type: 0x0003 Invoking createStream RTMP_ClientPacket, received: invoke 29 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 2.00> Property: NULL Property: <Name: no-name., NUMBER: 1.00> (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <createStream> SendPlay, seekTime=0, stopTime=0, sending play: vrm_foot_live Invoking play sending ctrl. type: 0x0003 HandleCtrl, received ctrl. type: 0, len: 6 HandleCtrl, Stream Begin 1 RTMP_ClientPacket, received: invoke 96 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Play.Start> Property: <Name: description, STRING: Start live> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Play.Start RTMP_ClientPacket, received: notify 24 bytes (object begin) Property: <Name: no-name., STRING: |RtmpSampleAccess> Property: <Name: no-name., BOOLEAN: TRUE> Property: <Name: no-name., BOOLEAN: TRUE> (object end) RTMP_ClientPacket, received: invoke 110 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Play.PublishNotify> Property: <Name: description, STRING: Start publishing> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Play.PublishNotify RTMP_ClientPacket, received: notify 387 bytes (object begin) Property: <Name: no-name., STRING: onMetaData> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: Server, STRING: NGINX RTMP (github.com/arut/nginx-rtmp-module)> Property: <Name: width, NUMBER: 1280.00> Property: <Name: height, NUMBER: 720.00> Property: <Name: displayWidth, NUMBER: 1280.00> Property: <Name: displayHeight, NUMBER: 720.00> Property: <Name: duration, NUMBER: 0.00> Property: <Name: framerate, NUMBER: 25.00> Property: <Name: fps, NUMBER: 25.00> Property: <Name: videodatarate, NUMBER: 976.00> Property: <Name: videocodecid, NUMBER: 7.00> Property: <Name: audiodatarate, NUMBER: 62.00> Property: <Name: audiocodecid, NUMBER: 10.00> Property: <Name: profile, STRING: > Property: <Name: level, STRING: > (object end) (object end) Metadata: Server NGINX RTMP (github.com/arut/nginx-rtmp-module) width 1280.00 height 720.00 displayWidth 1280.00 displayHeight 720.00 duration 0.00 framerate 25.00 fps 25.00 videodatarate 976.00 videocodecid 7.00 audiodatarate 62.00 audiocodecid 10.00 [h264 @ 0x138b040] Reinit context to 1280x720, pix_fmt: yuv420p Input #0, live_flv, from 'rtmp://xxxxxxxx': Metadata: Server : NGINX RTMP (github.com/arut/nginx-rtmp-module) displayWidth : 1280 displayHeight : 720 fps : 25 profile : level : Duration: 00:00:00.00, start: 100110.018000, bitrate: N/A Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720 [SAR 1:1 DAR 16:9], 999 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 63 kb/s Parsing... Parsed protocol: 0 Parsed host : 127.0.0.1 Parsed app : vip RTMP_Connect1, ... connected, handshaking HandShake: Type Answer : 03 HandShake: Server Uptime : 134227824 HandShake: FMS Version : 0.0.0.0 HandShake: Handshaking finished.... RTMP_Connect1, handshaked Invoking connect HandleServerBW: server BW = 5000000 HandleClientBW: client BW = 5000000 2 HandleChangeChunkSize, received: chunk size change to 1024 RTMP_ClientPacket, received: invoke 190 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 1.00> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: fmsVer, STRING: FMS/3,0,1,123> Property: <Name: capabilities, NUMBER: 31.00> (object end) Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetConnection.Connect.Success> Property: <Name: description, STRING: Connection succeeded.> Property: <Name: objectEncoding, NUMBER: 0.00> (object end) (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <connect> Invoking releaseStream Invoking FCPublish Invoking createStream RTMP_ClientPacket, received: invoke 29 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 4.00> Property: NULL Property: <Name: no-name., NUMBER: 1.00> (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <createStream> Invoking publish RTMP_ClientPacket, received: invoke 105 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Publish.Start> Property: <Name: description, STRING: Start publishing> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Publish.Start Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_qsv)) Stream #0:1 -> #0:1 (aac (native) -> aac (native)) Press [q] to stop, [?] for help [graph_1_in_0_1 @ 0x168c680] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3 [h264 @ 0x14b4f40] Reinit context to 1280x720, pix_fmt: yuv420p [graph 0 input from stream 0:0 @ 0x139b400] w:1280 h:720 pixfmt:yuv420p tb:1/1000 fr:25/1 sar:1/1 sws_param:flags=2 [auto_scaler_0 @ 0x13f6600] w:iw h:ih flags:'bicubic' interl:0 [format @ 0x148d5c0] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format' [auto_scaler_0 @ 0x13f6600] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:1280 h:720 fmt:nv12 sar:1/1 flags:0x4 [h264_qsv @ 0x13903c0] Initialized an internal MFX session using hardware accelerated implementation [h264_qsv @ 0x13903c0] Using the VBR with lookahead (LA) ratecontrol method [h264_qsv @ 0x13903c0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode. Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height Invoking FCUnpublish Invoking deleteStream [aac @ 0x14854c0] Qavg: 62580.895 [aac @ 0x14854c0] 2 frames left in the queue on closing Invoking deleteStream Conversion failed!
comment:18 by , 7 years ago
I've tried to reproduce it on ffmpeg-4.0 branch:
./ffmpeg -loglevel verbose -init_hw_device qsv=hw -i /samba/anonymous/Videos/bbb_sunflower_1080p_30fps_normal.mp4 -c:v h264_qsv -b:v 1000k test.mp4
The transcoding pipeline didn't crash, but reported "Encoder will work with partial HW acceleration, Warning in encoder initialization: partial acceleration (4)". The transcoding speed was very slow (about 20 fps), and it shows the gpu usage is zero when use metrics_monitor to check GPU usage.
It means software path was used instead of GPU accelerated.
comment:19 by , 7 years ago
More detail:
OS info:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.3.1611 (Core)
Release: 7.3.1611
Codename: Core
MSDK version: MediaServerStudioEssentials2017R3 (API version 1.23)
FFmpeg log (see the attachment)
Check GPU usage with metrics_monitor
./metrics_monitor RENDER usage: 0.00, VIDEO usage: 0.00, VIDEO_E usage: 0.00 VIDEO2 usage: 0.00 GT Freq: 300.00
by , 7 years ago
Attachment: | ffmpeg-20180424-153510.log added |
---|
comment:20 by , 7 years ago
Cc: | added |
---|
comment:21 by , 7 years ago
Not sure why it works well on Mark's side as comments #10 (Maybe it just broken on Linux but works well on Linux? I can understand "-init_hw_device" is not needed on Windows since the commit 1f26a231bb065276cd80ce02957c759f3197edfa removing ff_qsv_set_display_handle() has no impact on Windows).
But on Linux, passing a decoded frame by software decoder to qsv encoder directly is broken on FFmpeg 4.0 (this is the problem of this ticket). Passing a qsv decoded frame to a software encoder is also broken.(Reproduce: ffmpeg -c:v h264_qsv -i bbb_sunflower_1080p_30fps_normal.h264 -vframes 100 -c:v libx264 test.h264). These paths are supported on ffmpeg-3.4.
On FFmpeg 4.0, looks like we must use "hwupload" or "hwdownload" for these cases.
An example of this issue, an alternative way without palich's patch is using hwupload:
./ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i input.mp4 -vframes 100 -vf "hwupload=extra_hw_frames=64, format=qsv" -c:v h264_qsv out.mp4
comment:22 by , 5 years ago
Cc: | added |
---|
Double confirmed that it works on windows and this issue occurs on linux only in ffmpeg qsv decode, and could be reproducde with any clips(H264/H265).
And this issue could be split into two parts:
- which part introduces this regression?
- why it works on windows but fails on linux?
Analysis:
1.
This error may be relevant with FactoryCORE::CreateCORE and VAAPIVideoCORE::SetHandle.
On Linux, the created CORE is used in MFXVideoDECODE_Init/MFX_Utility::CheckVideoParam without calling SetHandle.
However, Application shall call SetHandle on Linux, but it wasn't called in Linux pipeline currently after FactoryCORE::CreateCORE was called.(in MFXInit() inside ff_qsv_process_data)
2.And the reason it works for windows may be:
"On the past we intentionally didn't implement initialization of vaDisplay inside MSDK library on Linux because on Linux there were many back-ends(X11, Wayland, headless drm, etc) and that time we didn't want to bring this logic inside MediaSDK."
comment:23 by , 5 years ago
Before commit http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=b0cd14fb1dab4b044f7fe6b53ac635409849de77#patch3,
- if (ist->hwaccel_id != HWACCEL_NONE) - type = hw_device_match_type_by_hwaccel(ist->hwaccel_id); - else - type = hw_device_match_type_in_name(ist->dec->name); - if (type != AV_HWDEVICE_TYPE_NONE) { dev = hw_device_get_by_type(type);
dev could be got through dec->name and then calls ff_qsv_init_session_device to CreateCORE/SetHandle and make sure the decoder works well.
However, after this commit, dev could not be got in hw_device_match_by_codec
dev = hw_device_match_by_codec(ist->dec);
because AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX is not configured as a method in ff_qsv_hw_configs.
+ for (i = 0;; i++) { + config = avcodec_get_hw_config(codec, i); + if (!config) + return NULL; + if (!(config->methods & AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX)) + continue; + dev = hw_device_get_by_type(config->device_type); + if (dev) + return dev; }
As a consequence, ff_qsv_init_internal_session was called to CreateCORE only without SetHandle.
And it leads to the decode failure.
Also as Zhong has mentioned, the removal of ff_qsv_set_display_handle() in ff_qsv_init_internal_session is one of the reasons as well.
comment:24 by , 5 years ago
https://patchwork.ffmpeg.org/patch/14898/ try to fix it. Please help to verify.
comment:25 by , 5 years ago
Resolution: | → fixed |
---|---|
Status: | open → closed |
Patch merged as 74007dd86a87289a075926704fae5bd8ef313bb5
Please find the commit that introduced the regression.