crash when encode with libx265

[Viacheslav]

Summary of the bug: when i try encode have several errors and ffmpeg crash

malloc_consolidate(): invalid chunk size or corrupted double-linked or double free or corruption or corrupted size vs. prev_size or Segmentation fault

How to reproduce:

ffmpeg -i ./sample_caused_crash_264to265.mkv -map 0 -c:v libx265 -preset medium -crf 23 ./output.mkv

sample file uploaded to videolan

ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
configuration: --enable-nonfree --enable-gpl --enable-openssl --enable-libdav1d --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libvpx --enable-libxvid --enable-libvidstab --enable-libaom --enable-libzimg --enable-lv2 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libtheora --enable-libfdk-aac --enable-libwebp --enable-libfreetype --enable-libsrt --enable-libzmq --disable-ffnvcodec --enable-amf --disable-shared --enable-pthreads --enable-static --enable-version3 --extra-cflags='-I/app/workspace/include -I/app/workspace/include/lilv-0' --extra-ldexeflags= --extra-ldflags=-L/app/workspace/lib --extra-libs='-ldl -lpthread -lm -lz' --pkgconfigdir=/app/workspace/lib/pkgconfig --pkg-config-flags=--static --prefix=/app/workspace --extra-version=
libavutil      58. 29.100 / 58. 29.100
libavcodec     60. 31.102 / 60. 31.102
libavformat    60. 16.100 / 60. 16.100
libavdevice    60.  3.100 / 60.  3.100
libavfilter     9. 12.100 /  9. 12.100
libswscale      7.  5.100 /  7.  5.100
libswresample   4. 12.100 /  4. 12.100
libpostproc    57.  3.100 / 57.  3.100

[Viacheslav]

similar ​https://trac.ffmpeg.org/ticket/10906
7.0 also crashing

[Henrique Andrade]

Fwiw, I also see a similar crash when encoding directly from HDHomeRun encoder as follows:

ffmpeg -y -hide_banner -threads 1 -loglevel error -i 'http://hdhr2:5004/auto/v2.1?duration=1800' -s 1920x1080 -preset ultrafast -c:v libx265 -crf 23 -max_muxing_queue_size 9999 -c:a mp3 -strict -2 /tmp/en.mkv
[mpeg2video @ 0x5d5115f4dd40] Invalid frame dimensions 0x0.
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 13.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 2 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : dia / 57 / 0 / 2
x265 [info]: Lookahead / bframes / badapt        : 5 / 3 / 0
x265 [info]: b-pyramid / weightp / weightb       : 1 / 0 / 0
x265 [info]: References / ref-limit  cu / depth  : 1 / off / off
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-23.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 tmvp fast-intra
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock
corrupted size vs. prev_size

I tested and repro'ed this using the statically built versions from 4.3.2 to 7.0.1 (Linux builds from ​https://www.ffmpeg.org/download.html#build-linux). As well as the natively shipped version on Ubuntu 24.02, which is:

ffmpeg version 6.1.1-3ubuntu5 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13 (Ubuntu 13.2.0-23ubuntu3)

[Viacheslav]

UPD: Master branch encode normally my sample file