ffv1enc: Encoding >5K content fails with "Invalid minimum required packet size"

[Chris]

Summary of the bug:
If a input resolution greater than 5K (5120 × 2880) is passed through ffmpeg to encode with FFV1, it will fail encoding with "Invalid minimum required packet size". For this example, I will use 5632x3168 as the resolution. When that resolution is passed, it will create a maxsize value of 2640658432 (5632 * 3168 * 37 * 4 + 16384) and then passes that value directly to ff_alloc_packet2, without finding out the actual size of the packet. ff_alloc_packet2 then fails the check for the maximum packet size of MAX_INT - 32 because it's checking the maximum size allowed by the resolution instead of the actual size.

Here is how to re-produce the issue and the output provided:

% ffmpeg -f lavfi -i testsrc=size=5632x3168 -pix_fmt yuv420p -t 60 -vcodec ffv1 test.avi
ffmpeg version N-82794-g3ab1311 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 42.100 / 55. 42.100
  libavcodec     57. 67.100 / 57. 67.100
  libavformat    57. 58.102 / 57. 58.102
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 68.100 /  6. 68.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, lavfi, from 'testsrc=size=5632x3168':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 5632x3168 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
File 'test.avi' already exists. Overwrite ? [y/N] y
Output #0, avi, to 'test.avi':
  Metadata:
    ISFT            : Lavf57.58.102
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 5632x3168 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.67.100 ffv1
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> ffv1 (native))
Press [q] to stop, [?] for help
[ffv1 @ 00000000024a5840] Invalid minimum required packet size 2640658432 (max allowed is 2147483615)
Video encoding failed
Conversion failed!

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Michael Niedermayer]

ffv1 version 4 will have a smaller worst case size, well it actually already does but the bitstream for ver 4 is not final yet

[Michael Niedermayer]

Fixed in 38a7834bbb24ef62466b076715e0add60e1d6962

[Michael Niedermayer]

undo unintended changes