"Lossless encoding not supported" error occurs using NVIDIA Video Codec SDK 10.0

[bombur33]

Summary of the bug:
When using h264 nvenc, a "Lossless encoding not supported" error occurs using NVIDIA Video Codec SDK 10.0 on certain NVIDIA cards despite using lossy settings. In this case, it's a GTX 960 (Maxwell), which doesn't have lossless capability. When lossy settings are used, it shouldn't matter that the card doesn't support lossless.

The problem seems to trace back from around line 361 of libavcodec/nvenc.c that checks ctx->preset >= PRESET_LOSSLESS_DEFAULT. The new SDK 10 presets (p1-p7) seem to evaluate greater than PRESET_LOSSLESS_DEFAULT despite the new -tune parameter not being set to lossless. In SDK 10, the -tune parameter controls lossless rather than the preset. See the preset migration guide:
​https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-preset-migration-guide/#h264-preset-migration-table
How to reproduce:

% ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input -c:v h264_nvenc -preset:v p1 -tune:v hq -rc:v vbr -c:a copy -y output.mp4
ffmpeg version
built on 2021-03-24

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[bombur33]

run log

[bombur33]

I should mention, the problem occurs when using the new p1-p7 presets added in SDK 10 on GPUs that don't support lossless. I've run the same test successfully on a newer NVIDIA GPU that supports lossless.

[Balling]

Replying to bombur33:

I should mention, the problem occurs when using the new p1-p7 presets added in SDK 10 on GPUs that don't support lossless. I've run the same test successfully on a newer NVIDIA GPU that supports lossless.

Well, obviously. So the patch is just to add < PRESET_P1 and maybe the same about ctx->tuning_info in #ifdef NVENC_HAVE_NEW_PARAMETERS.

But yeah, that is because ret will be < 0!

To oromit: I tested it on 2080 Ti it works great there because lossless is supported and we just do not return. So I checked with his command... It is not lossless... While -tune:v lossless will be lossless...

[Balling]

Looks like there is also this comment. ​https://github.com/FFmpeg/FFmpeg/commit/cde3c080338154cc73103939f6ab26ae5bb718cd#comments

[Balling]

Can you fix this ASAP? I proposed a patch but not sure about whether we should limit/check ctx->tuning_info too.

[Balling]

Fixed in 988f2e9eb063db7c1a678729f58aab6eba59a55b.