VPP_QSV filter don't work as expected

[thx4ever]

Try to convert a 8bit AVC to 10bit HEVC using Intel HEVC Plugin software encoder

C:\FFMPEG>ffmpeg -v verbose -c:v h264_qsv -i C:\Users\thx4ever\Downloads\sample_AVC.264 -vf vpp_qsv,format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryslow -g 240 -refs:v 6 -bf:v 3 -flags -cgop -gpb 0 -q:v 18 -vframes 5 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v hevc_metadata=colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 C:\Users\thx4ever\Desktop\sample_HEVC.hevc
ffmpeg version 4.1.git Copyright (c) 2000-2019 the FFmpeg developers

built with msvc
configuration: --enable-gpl --enable-version3 --enable-bzlib --enable-iconv --enable-lzma --enable-sdl2 --enable-zlib --enable-avisynth --enable-libmp3lame --enable-libvorbis --enable-libspeex --enable-libopus --enable-libilbc --enable-libtheora --enable-libx264 --enable-libx265 --enable-libxvid --enable-libvpx --enable-libgme --enable-libmodplug --enable-libsoxr --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-libass --enable-libxml2 --enable-gnutls --disable-schannel --enable-gcrypt --enable-libssh --enable-libcdio --enable-libbluray --enable-opengl --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-amf --toolchain=msvc
libavutil 56. 30.100 / 56. 30.100
libavcodec 58. 53.101 / 58. 53.101
libavformat 58. 28.102 / 58. 28.102
libavdevice 58. 7.100 / 58. 7.100
libavfilter 7. 56.101 / 7. 56.101
libswscale 5. 4.101 / 5. 4.101
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100

[h264 @ 000001B309F26000] Reinit context to 1280x720, pix_fmt: yuv420p
[h264 @ 000001B309F12140] max_analyze_duration 5000000 reached at 5000000 microseconds st:0
Input #0, h264, from 'C:\Users\thx4ever\Downloads\sample_AVC.264':

Duration: N/A, bitrate: N/A

Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc

[AVBSFContext @ 000001B30C5AA780] The input looks like it is Annex B already
Stream mapping:

Stream #0:0 -> #0:0 (h264 (h264_qsv) -> hevc (hevc_qsv))

Press [q] to stop, ? for help
[h264_qsv @ 000001B30C00EB00] Initialized an internal MFX session using hardware accelerated implementation
[graph 0 input from stream 0:0 @ 000001B30C613740] w:1280 h:720 pixfmt:nv12 tb:1/1200000 fr:25/1 sar:0/1 sws_param:flags=2
[Parsed_vpp_qsv_0 @ 000001B30C613040] qsv vpp pass through mode.
[hevc_qsv @ 000001B30C650500] Initialized an internal MFX session using hardware accelerated implementation
[hevc_qsv @ 000001B30C650500] Using the constant quantization parameter (CQP) ratecontrol method
[hevc_qsv @ 000001B30C650500] profile: main10; level: 31
[hevc_qsv @ 000001B30C650500] GopPicSize: 240; GopRefDist: 4; GopOptFlag: ; IdrInterval: 1
[hevc_qsv @ 000001B30C650500] TargetUsage: 1; RateControlMethod: CQP
[hevc_qsv @ 000001B30C650500] QPI: 14; QPP: 18; QPB: 23
[hevc_qsv @ 000001B30C650500] NumSlice: 1; NumRefFrame: 6
[hevc_qsv @ 000001B30C650500] RateDistortionOpt: unknown
[hevc_qsv @ 000001B30C650500] RecoveryPointSEI: unknown IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0
[hevc_qsv @ 000001B30C650500] MaxFrameSize: 0; MaxSliceSize: 0;
[hevc_qsv @ 000001B30C650500] BitrateLimit: unknown; MBBRC: ON; ExtBRC: unknown
[hevc_qsv @ 000001B30C650500] Trellis: auto
[hevc_qsv @ 000001B30C650500] VDENC: OFF
[hevc_qsv @ 000001B30C650500] RepeatPPS: unknown; NumMbPerSlice: 0; LookAheadDS: unknown
[hevc_qsv @ 000001B30C650500] AdaptiveI: OFF; AdaptiveB: unknown; BRefType: auto
[hevc_qsv @ 000001B30C650500] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0
[hevc_qsv @ 000001B30C650500] GPB: OFF
[hevc_qsv @ 000001B30C650500] FrameRateExtD: 1; FrameRateExtN: 25
Output #0, hevc, to 'C:\Users\thx4ever\Desktop\sample_HEVC.hevc':

Metadata:

encoder : Lavf58.28.102
Stream #0:0: Video: hevc (hevc_qsv), 1 reference frame, p010le(bt709, left), 1280x720, q=2-31, 1000 kb/s, 25 fps, 25 tbn, 25 tbc
Metadata:

encoder : Lavc58.53.101 hevc_qsv

Side data:

cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: -1

No more output streams to write to, finishing.
frame= 5 fps=2.7 q=-0.0 Lsize= 4580kB time=00:00:00.60 bitrate=62533.0kbits/s speed=0.324x
video:4580kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Input file #0 (C:\Users\thx4ever\Downloads\sample_AVC.264):

Input stream #0:0 (video): 17 packets read (242428 bytes); 6 frames decoded;
Total: 17 packets (242428 bytes) demuxed

Output file #0 (C:\Users\thx4ever\Desktop\sample_HEVC.hevc):

Output stream #0:0 (video): 5 frames encoded; 5 packets muxed (4689978 bytes);
Total: 5 packets (4689978 bytes) muxed

[AVIOContext @ 000001B309ED34C0] Statistics: 0 seeks, 18 writeouts
[AVIOContext @ 000001B309ECB340] Statistics: 1540096 bytes read, 0 seeks

If you take a look at info it says qsv vpp pass through mode
so it should convert to P010le pix_fmt to encode 8 bit AVC to 10 bit HEVC

[thx4ever]

HEVC Encoded bitstream using ffmpeg with Intel HEVC Software encoder

[thx4ever]

Source file AVC

[Zhong,Li]

Could you please highlight what is your expection and what is the real output?

[thx4ever]

I expect that it will convert the pix_fmt 8 bit to 10 bit using VPP to be able to use HEVC 10bit encoding, but right now the real output are garbage like i have added in attachment sample_HEVC, what it needs to do its convert the sample_AVC that are 8bit to 10 bit using Intel QSV VPP

[Zhong,Li]

Haven't tested it on windows with qsv HEVC software encoding. But I can reproduce your issue on Linux with HEVC HW encoding:

ffmpeg -hwaccel qsv -v verbose -c:v h264_qsv -i tikect_8065/sample_AVC.264 -vf vpp_qsv=format=p010le -c:v hevc_qsv -load_plugin hevc_hw -profile:v main10 -preset:v veryslow -g 240 -refs:v 6 -bf:v 3 -flags -cgop -gpb 0 -q:v 18 -vframes 5 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v hevc_metadata=colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 sample_HEVC.hevc

Transcoding failed with "qsv vpp pass through mode" message.

[Zhong,Li]

​https://patchwork.ffmpeg.org/patch/14522/ has been sent out to fix this issue.

[thx4ever]

Replying to lizhong1008:

​https://patchwork.ffmpeg.org/patch/14522/ has been sent out to fix this issue.

I still can reproduce same issue on Windows with your patch, my command line options used

ffmpeg -v verbose -hide_banner -hwaccel qsv -c:v h264_qsv -i sample_AVC.264 -vf scale_qsv=1280:720,vpp_qsv=format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v slower -b:v 2280k -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v hevc_metadata=colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 -gpb 0 -vframes 5 sample_encoded_hevc_ticket#8065.hevc
[h264 @ 000002A4ED03D900] Reinit context to 1280x720, pix_fmt: yuv420p
[h264 @ 000002A4ED028A00] max_analyze_duration 5000000 reached at 5000000 microseconds st:0
Input #0, h264, from 'C:\Users\thx4ever\Downloads\sample_AVC.264':

Duration: N/A, bitrate: N/A

Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc

[AVBSFContext @ 000002A4EED98340] The input looks like it is Annex B already
Stream mapping:

Stream #0:0 -> #0:0 (h264 (h264_qsv) -> hevc (hevc_qsv))

Press [q] to stop, ? for help
[AVHWDeviceContext @ 000002A4EF461640] Using D3D9Ex device.
[AVHWDeviceContext @ 000002A4EF461140] Initialize MFX session: API version is 1.27, implementation version is 1.29
[AVHWDeviceContext @ 000002A4EF461140] MFX compile/runtime API: 1.27/1.29
[graph 0 input from stream 0:0 @ 000002A4EEE01DC0] w:1280 h:720 pixfmt:qsv tb:1/1200000 fr:25/1 sar:0/1 sws_param:flags=2
[Parsed_scale_qsv_0 @ 000002A4EEFBA500] Scaling mode: 0
[Parsed_scale_qsv_0 @ 000002A4EEFBA500] w:1280 h:720 -> w:1280 h:720
[hevc_qsv @ 000002A4ED0CEB00] Using the average variable bitrate (AVBR) ratecontrol method
[hevc_qsv @ 000002A4ED0CEB00] profile: main10; level: 31
[hevc_qsv @ 000002A4ED0CEB00] GopPicSize: 248; GopRefDist: 9; GopOptFlag: closed ; IdrInterval: 1
[hevc_qsv @ 000002A4ED0CEB00] TargetUsage: 2; RateControlMethod: AVBR
[hevc_qsv @ 000002A4ED0CEB00] TargetKbps: 2280; Accuracy: 0; Convergence: 0; BRCParamMultiplier: 1
[hevc_qsv @ 000002A4ED0CEB00] NumSlice: 1; NumRefFrame: 4
[hevc_qsv @ 000002A4ED0CEB00] RateDistortionOpt: unknown
[hevc_qsv @ 000002A4ED0CEB00] RecoveryPointSEI: unknown IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0
[hevc_qsv @ 000002A4ED0CEB00] MaxFrameSize: 0; MaxSliceSize: 0;
[hevc_qsv @ 000002A4ED0CEB00] BitrateLimit: unknown; MBBRC: ON; ExtBRC: unknown
[hevc_qsv @ 000002A4ED0CEB00] Trellis: auto
[hevc_qsv @ 000002A4ED0CEB00] VDENC: OFF
[hevc_qsv @ 000002A4ED0CEB00] RepeatPPS: unknown; NumMbPerSlice: 0; LookAheadDS: unknown
[hevc_qsv @ 000002A4ED0CEB00] AdaptiveI: ON; AdaptiveB: unknown; BRefType: auto
[hevc_qsv @ 000002A4ED0CEB00] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0
[hevc_qsv @ 000002A4ED0CEB00] GPB: OFF
[hevc_qsv @ 000002A4ED0CEB00] FrameRateExtD: 1; FrameRateExtN: 25
Output #0, hevc, to 'C:\Users\thx4ever\Desktop\sample_encoded_hevc_ticket#8065.hevc':

Metadata:

encoder : Lavf58.30.100
Stream #0:0: Video: hevc (hevc_qsv), 1 reference frame, qsv(bt709, left), 1280x720, q=2-31, 2280 kb/s, 25 fps, 25 tbn, 25 tbc
Metadata:

encoder : Lavc58.55.100 hevc_qsv

Side data:

cpb: bitrate max/min/avg: 0/0/2280000 buffer size: 0 vbv_delay: 18446744073709551615

No more output streams to write to, finishing.
frame= 5 fps=4.5 q=-0.0 Lsize= 2972kB time=00:00:00.52 bitrate=46815.9kbits/s speed=0.468x
video:2972kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Input file #0 (C:\Users\thx4ever\Downloads\sample_AVC.264):

Input stream #0:0 (video): 17 packets read (242428 bytes); 6 frames decoded;
Total: 17 packets (242428 bytes) demuxed

Output file #0 (C:\Users\thx4ever\Desktop\sample_encoded_hevc_ticket#8065.hevc):

Output stream #0:0 (video): 5 frames encoded; 5 packets muxed (3043031 bytes);
Total: 5 packets (3043031 bytes) muxed

[AVIOContext @ 000002A4ECFDB380] Statistics: 0 seeks, 12 writeouts
[AVIOContext @ 000002A4ECFD9A40] Statistics: 1540096 bytes read, 0 seeks

first the bitrate of output goes crazy and the final encoded vídeo is garbage of colors

[Zhong,Li]

Replying to thx4ever:

Replying to lizhong1008:

​https://patchwork.ffmpeg.org/patch/14522/ has been sent out to fix this issue.

I still can reproduce same issue on Windows with your patch, my command line options used

It is not same issue right now. The "pass through mode" message goes away, and you changed your command line: add scaling and BRC modes: CQP->AVBR

Some questions:

1. Have you tried HEVC HW encoder,what is the result?
2. Why you change BRC mode from CQP to AVBR and add scaling? Dose CQP mode can work?
3. Have you tried HEVC SW encoder with MSDK sample encoder?
4. Try to figure out it is an qsv vpp issue or encoder issue. You can disable encoder and check the output of vpp.

[thx4ever]

Replying to lizhong1008:

Replying to thx4ever:

Replying to lizhong1008:

​https://patchwork.ffmpeg.org/patch/14522/ has been sent out to fix this issue.

I still can reproduce same issue on Windows with your patch, my command line options used

It is not same issue right now. The "pass through mode" message goes away, and you changed your command line: add scaling and BRC modes: CQP->AVBR

Some questions:

1. Have you tried HEVC HW encoder,what is the result?
2. Why you change BRC mode from CQP to AVBR and add scaling? Dose CQP mode can work?
3. Have you tried HEVC SW encoder with MSDK sample encoder?
4. Try to figure out it is an qsv vpp issue or encoder issue. You can disable encoder and check the output of vpp.

1 - Yes i have tried HEVC HW Encoder and it Works but when try software encoder the final result are garbage of colors, it seems something is wrong with ffmpeg
2- Sorry for that but doenst matter what BRC i use, CQP-AVB-VBR on HW it Works but software is garbage, can u address this bug to ffmpeg developers to fix this issue? This Works on sample_multi_transcode on Windows 10 Pro x64 with git Media SDK
3- Yes sample encoder Works but i use alot ffmpeg right now
4- OK i will try and send feedback to you

[thx4ever]

I have checked it Works but when i add -pix_fmt p010le the final result is incorrect, i think the problem are in ffmpeg color conversion from 8bit to 10bit, can u look for me this problem?

Regards

thx4ever

[Zhong,Li]

Replying to thx4ever:

1 - Yes i have tried HEVC HW Encoder and it Works but when try software encoder the final result are garbage of colors, it seems something is wrong with ffmpeg
2- Sorry for that but doenst matter what BRC i use, CQP-AVB-VBR on HW it Works but software is garbage, can u address this bug to ffmpeg developers to fix this issue? This Works on sample_multi_transcode on Windows 10 Pro x64 with git Media SDK
3- Yes sample encoder Works but i use alot ffmpeg right now

Could you please share your sample encoder and sample multi transcode command line?

I checked MediaServerStudioProfessional2017 release guide, it is said that:

MAIN10 profile encoding/decoding and range extensions profile encoding can
be done only with system memory at input and output on linux (SDK/driver
limitation for video memory frames allocation)

Thus means probally you should remove the "-hwaccel qsv" to make it is working in sytstem memory:

ffmpeg-v verbose -c:v h264_qsv -i tikect_8065/sample_AVC.264 -vf vpp_qsv=format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryslow -g 240 -refs:v 6 -bf:v 3 -flags -cgop -gpb 0 -q:v 18 -vframes 5 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v hevc_metadata=colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 sample_HEVC.hevc

Please tried it and update the status.

[thx4ever]

Replying to lizhong1008:

Replying to thx4ever:

1 - Yes i have tried HEVC HW Encoder and it Works but when try software encoder the final result are garbage of colors, it seems something is wrong with ffmpeg
2- Sorry for that but doenst matter what BRC i use, CQP-AVB-VBR on HW it Works but software is garbage, can u address this bug to ffmpeg developers to fix this issue? This Works on sample_multi_transcode on Windows 10 Pro x64 with git Media SDK
3- Yes sample encoder Works but i use alot ffmpeg right now

Could you please share your sample encoder and sample multi transcode command line?

I checked MediaServerStudioProfessional2017 release guide, it is said that:

MAIN10 profile encoding/decoding and range extensions profile encoding can
be done only with system memory at input and output on linux (SDK/driver
limitation for video memory frames allocation)

Thus means probally you should remove the "-hwaccel qsv" to make it is working in sytstem memory:

ffmpeg-v verbose -c:v h264_qsv -i tikect_8065/sample_AVC.264 -vf vpp_qsv=format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryslow -g 240 -refs:v 6 -bf:v 3 -flags -cgop -gpb 0 -q:v 18 -vframes 5 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v hevc_metadata=colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 sample_HEVC.hevc

Please tried it and update the status.

I tried and dont work, check this problem No hw context provided if i remove hwaccel qsv option, maybe we need a option in ffmpeg to work in system memory wit hwaccel qsv option on

ffmpeg -v verbose -hide_banner -c:v h264_qsv -i "C:\Users\thx4ever\Downloads\sample_AVC (2).264" -vf vpp_qsv=format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryslow -g 240 -refs:v 6 -bf:v 3 -flags -cgop -gpb 0 Sample.hevc
[h264 @ 000002A572E37EC0] Reinit context to 1280x720, pix_fmt: yuv420p
[h264 @ 000002A572E23C40] max_analyze_duration 5000000 reached at 5000000 microseconds st:0
Input #0, h264, from 'C:\Users\thx4ever\Downloads\sample_AVC (2).264':

Duration: N/A, bitrate: N/A

Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc

File 'Sample.hevc' already exists. Overwrite ? [y/N] y
[AVBSFContext @ 000002A574768D80] The input looks like it is Annex B already
Stream mapping:

Stream #0:0 -> #0:0 (h264 (h264_qsv) -> hevc (hevc_qsv))

Press [q] to stop, ? for help
[h264_qsv @ 000002A575188680] Initialized an internal MFX session using hardware accelerated implementation
[graph 0 input from stream 0:0 @ 000002A574D6C040] w:1280 h:720 pixfmt:nv12 tb:1/1200000 fr:25/1 sar:0/1 sws_param:flags=2
[Parsed_vpp_qsv_0 @ 000002A574D6C240] No hw context provided.
[Parsed_vpp_qsv_0 @ 000002A574D6C240] Failed to configure output pad on Parsed_vpp_qsv_0
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
[AVIOContext @ 000002A57479EE80] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 000002A572DFE540] Statistics: 1540096 bytes read, 0 seeks
Conversion failed!

Here are my command line options for sample_multi_transcode

sample_multi_transcode -i::h264 "C:\Users\thx4ever\Desktop\Sample.264 -o::h265 C:\Users\thx4ever\Desktop\sample.hevc -hw -async 10 -sys -pe hevce_sw -cqp -qpi 20 -qpp 22 -qpb 24 -u quality -l 1 -gop_size 240 -dist 4 -num_ref 6 -bref -bpyr -gpb:off -TransformSkip:on -ec::nv12 -dc::nv12 -w 1920 -h 1080 -WeightedPred::default -WeightedBiPred::default -amfs:on -pp

I'm using latest Intel Media SDK from GitHub

This pull request [samples] Correct vpp output bit depth according to FourCC #1416 from Intel Media SDK git did the trick for sample_multi_transcode working in hardware to convert from 8 bit to 10 bit, check the pull request from Intel Media SDK

[Zhong,Li]

Replying to thx4ever:

I tried and dont work, check this problem No hw context provided if i remove hwaccel qsv option, maybe we need a option in ffmpeg to work in system memory wit hwaccel qsv option on

Probably you have two options:

1. remove "-hwaccel qsv" and add "-filter_hw_device".
2. keep "-hwaccel qsv " and use "hwdownload" See examples of ​http://trac.ffmpeg.org/wiki/Hardware/QuickSync

I'm using latest Intel Media SDK from GitHub

I don't think github MSDK can support Windows. Is there anything wrong?

[thx4ever]

The only way to get ffmpeg working are this command options but the final Picture is garbage

ffmpeg -v verbose -hide_banner -hwaccel qsv -c:v h264_qsv -i "sample.264" -vf vpp_qsv=format=p010le,hwdownload,format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryfast C:\Users\thx4ever\Desktop\sample.hevc

Yes i can use GitHub MSDK, that is what i currently use

[Zhong,Li]

Replying to thx4ever:

The only way to get ffmpeg working are this command options but the final Picture is garbage

ffmpeg -v verbose -hide_banner -hwaccel qsv -c:v h264_qsv -i "sample.264" -vf vpp_qsv=format=p010le,hwdownload,format=p010le -c:v hevc_qsv -load_plugin hevc_sw -profile:v main10 -preset:v veryfast C:\Users\thx4ever\Desktop\sample.hevc

Yes i can use GitHub MSDK, that is what i currently use

​https://patchwork.ffmpeg.org/patch/14522/ has been merged since HEVC HW encoding can work now.
For HEVC SW encoding problem，probally you can file an issue on MSDK github too (but I can't find HEVC SW encoding can be supported by ​https://github.com/Intel-Media-SDK/MediaSDK).