vaapi_encode_check_config invalid free

[serafean]

Invalid free in vaapi_encode_check_config ( vaapi_encode_config_attributes in 3.3 branch - manually checked source code )
How to reproduce:

% MALLOC_CHECK_=2 ffmpeg -loglevel debug -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i Elephants_Dream_HD.avi -vf format=nv12,hwupload -map 0:0 -map 0:1 -y -f matroska -bf 0 -c:v h264_vaapi ~/test.mkv
3.2.4
built on Gentoo

(gdb) bt
#0  0x00007f0fbeb1eeb8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f0fbeb2044a in __GI_abort () at abort.c:89
#2  0x00007f0fbeb63890 in malloc_printerr (action=<optimized out>, str=0x7f0fbec55c27 "free(): invalid pointer", ptr=<optimized out>, 
    ar_ptr=<optimized out>) at malloc.c:5008
#3  0x00007f0fc01e88b2 in vaapi_encode_check_config (avctx=0x55e6562a3620) at src/libavcodec/vaapi_encode.c:1024
#4  ff_vaapi_encode_init (avctx=0x55e6562a3620, type=<optimized out>) at src/libavcodec/vaapi_encode.c:1076
#5  0x00007f0fc06353f0 in avcodec_open2 (avctx=0x55e6562a3620, codec=0x7f0fc1043cc0 <ff_h264_vaapi_encoder>, options=0x55e65624f888)
    at src/libavcodec/utils.c:1608
#6  0x000055e655ab58ca in init_output_stream (error_len=1024, error=0x7ffde9fa30e0 "", ost=0x55e65624f740) at src/ffmpeg.c:3024
#7  transcode_init () at src/ffmpeg.c:3482
#8  0x000055e655a98352 in transcode () at src/ffmpeg.c:4358
#9  main (argc=23, argv=0x7ffde9fa3a48) at src/ffmpeg.c:4592

The issue is that every "goto fail" tries to free both "profiles" and "entrypoints", when entrypoints might not even be allocated yet.

[jkqxz]

What is the actual error that you get here?

I don't see anything wrong with the entrypoints variable - it's initialised to NULL and then possibly overwritten by the return value of av_malloc_array(). Both of those are always valid things to pass to free().

[serafean]

Ah, my bad, I forgot about the free behaviour.

Crashes.
with MALLOC_CHECK_=1 segfault :

#0  _int_malloc (av=av@entry=0x7f8429308aa0 <main_arena>, bytes=bytes@entry=9) at malloc.c:3414
#1  0x00007f8428fe635b in malloc_check (sz=sz@entry=8, caller=caller@entry=0x0) at hooks.c:295
#2  0x00007f8428fe6d86 in realloc_check (oldmem=0x0, bytes=8, caller=<optimized out>) at hooks.c:355
#3  0x00007f8429a80c4f in av_frame_new_side_data (frame=frame@entry=0x563b5c49d7a0, type=AV_FRAME_DATA_MATRIXENCODING, size=4) at src/libavutil/frame.c:634
#4  0x00007f8429a80f40 in frame_copy_props (dst=dst@entry=0x563b5c49d7a0, src=0x563b5c4af0e0, force_copy=force_copy@entry=1) at src/libavutil/frame.c:339
#5  0x00007f8429a810a9 in av_frame_copy_props (dst=dst@entry=0x563b5c49d7a0, src=<optimized out>) at src/libavutil/frame.c:591
#6  0x00007f842be92141 in ff_filter_frame_needs_framing (frame=0x563b5c4af0e0, link=0x563b5c36c6c0) at src/libavfilter/avfilter.c:1162
#7  ff_filter_frame (link=0x563b5c36c6c0, frame=0x563b5c4af0e0) at src/libavfilter/avfilter.c:1230
#8  0x00007f842be90d9a in ff_filter_frame_framed (link=link@entry=0x563b5c36c540, frame=0x563b5c4af0e0) at src/libavfilter/avfilter.c:1134
#9  0x00007f842be920c0 in ff_filter_frame (link=0x563b5c36c540, frame=0x563b5c4af0e0) at src/libavfilter/avfilter.c:1232
#10 0x00007f842be90d9a in ff_filter_frame_framed (link=link@entry=0x563b5c36c2c0, frame=0x563b5c4af0e0) at src/libavfilter/avfilter.c:1134
#11 0x00007f842be920c0 in ff_filter_frame (link=link@entry=0x563b5c36c2c0, frame=0x563b5c4af0e0) at src/libavfilter/avfilter.c:1232
#12 0x00007f842be9658f in request_frame (link=0x563b5c36c2c0) at src/libavfilter/buffersrc.c:450
#13 0x00007f842be96244 in av_buffersrc_add_frame_internal (ctx=ctx@entry=0x563b5c36b840, frame=frame@entry=0x563b5c45db60, flags=flags@entry=4)
    at src/libavfilter/buffersrc.c:239
#14 0x00007f842be967ed in av_buffersrc_add_frame_flags (ctx=0x563b5c36b840, frame=0x563b5c45db60, flags=4) at src/libavfilter/buffersrc.c:164
#15 0x0000563b5c065a7d in decode_audio (got_output=0x7ffdb8e80c0c, pkt=0x7ffdb8e80c30, ist=0x563b5c342ee0) at src/ffmpeg.c:2164
#16 process_input_packet (ist=<optimized out>, pkt=0x7ffdb8e80f00, no_eof=0) at src/ffmpeg.c:2466
#17 0x0000563b5c046daa in process_input (file_index=<optimized out>) at src/ffmpeg.c:4245
#18 transcode_step () at src/ffmpeg.c:4333
#19 transcode () at src/ffmpeg.c:4387
#20 main (argc=<optimized out>, argv=<optimized out>) at src/ffmpeg.c:4592

Without MALLOC_CHECK_, sometimes SIGABRT:

*** Error in `ffmpeg': corrupted double-linked list: 0x000055ad35f86170 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x72a07)[0x7f784ac7da07]
/lib64/libc.so.6(+0x78866)[0x7f784ac83866]
/lib64/libc.so.6(+0x78c21)[0x7f784ac83c21]
/lib64/libc.so.6(+0x7a5d2)[0x7f784ac855d2]
/lib64/libc.so.6(__libc_malloc+0x63)[0x7f784ac876f3]
/usr/lib64/va/drivers/r600_drv_video.so(+0x210ca)[0x7f783bf140ca]
/usr/lib64/va/drivers/r600_drv_video.so(+0x22fa2)[0x7f783bf15fa2]
/usr/lib64/libavutil.so.55(+0x2072e)[0x7f784b72372e]
/usr/lib64/libavutil.so.55(+0x20a77)[0x7f784b723a77]
/usr/lib64/libavutil.so.55(av_hwframe_transfer_data+0xb7)[0x7f784b722ec7]
/usr/lib64/libavfilter.so.6(+0x10ef89)[0x7f784dba1f89]
/usr/lib64/libavfilter.so.6(+0x9dd9a)[0x7f784db30d9a]
/usr/lib64/libavfilter.so.6(+0x9f0c0)[0x7f784db320c0]
/usr/lib64/libavfilter.so.6(+0x9dd9a)[0x7f784db30d9a]
/usr/lib64/libavfilter.so.6(+0x9f0c0)[0x7f784db320c0]
/usr/lib64/libavfilter.so.6(+0x13d466)[0x7f784dbd0466]
/usr/lib64/libavfilter.so.6(+0x9dd9a)[0x7f784db30d9a]
/usr/lib64/libavfilter.so.6(+0x9f0c0)[0x7f784db320c0]
/usr/lib64/libavfilter.so.6(+0xa358f)[0x7f784db3658f]
/usr/lib64/libavfilter.so.6(+0xa3244)[0x7f784db36244]
/usr/lib64/libavfilter.so.6(av_buffersrc_add_frame_flags+0xb5)[0x7f784db367ed]
ffmpeg(+0x2bd05)[0x55ad34c4cd05]
ffmpeg(+0xcdaa)[0x55ad34c2ddaa]
/lib64/libc.so.6(__libc_start_main+0xfc)[0x7f784ac2b7cc]
ffmpeg(+0xea59)[0x55ad34c2fa59]

running it through Valgrind throws out SIGILL.

SIGSEGV in malloc is the most common.
MALLOC_CHECK_=2 is the only one which is deterministic, so I thought that could be it...
I'll try with ffmpeg 3.3 tomorrow.
Do you think it could be in the r600 driver?

[Carl Eugen Hoyos]

Please test current FFmpeg git head.

[serafean]

ffmpeg version : N-85962-g164e277
valgrind still raises SIGILL
MALLOC_CHECK_=2 still crashes with

(gdb) bt
#0  0x00007ffabdfc6eb8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffabdfc844a in __GI_abort () at abort.c:89
#2  0x00007ffabe00b890 in malloc_printerr (action=<optimized out>, str=0x7ffabe0fdc27 "free(): invalid pointer", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5008
#3  0x00007ffabf470411 in vaapi_encode_config_attributes (avctx=0x55ef16408aa0) at src/libavcodec/vaapi_encode.c:1107
#4  ff_vaapi_encode_init (avctx=0x55ef16408aa0) at src/libavcodec/vaapi_encode.c:1377
#5  0x00007ffabf8d78f9 in avcodec_open2 (avctx=0x55ef16408aa0, codec=0x7ffac0304ce0 <ff_h264_vaapi_encoder>, options=0x55ef163a67f8) at src/libavcodec/utils.c:1020
#6  0x000055ef142c8dde in init_output_stream (ost=<optimized out>, error=0x7fff49d94c20 "", error_len=1024) at src/ffmpeg.c:3438
#7  0x000055ef142ca9c1 in reap_filters (flush=0) at src/ffmpeg.c:1443
#8  0x000055ef142acd94 in transcode_step () at src/ffmpeg.c:4522
#9  transcode () at src/ffmpeg.c:4566
#10 main (argc=<optimized out>, argv=<optimized out>) at src/ffmpeg.c:477

The other two cases appear to work (albeit slowly, but that might be an r600 issue).

MALLOC_CHECK_=3 :

*** Error in `ffmpeg': free(): invalid pointer: 0x0000560c61308b20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x72a07)[0x7fea668a5a07]
/lib64/libc.so.6(+0x78866)[0x7fea668ab866]
/usr/lib64/libavcodec.so.57(+0xad411)[0x7fea67d10411]
/usr/lib64/libavcodec.so.57(avcodec_open2+0x879)[0x7fea681778f9]
ffmpeg(+0x28dde)[0x560c6094ddde]
ffmpeg(+0x2a9c1)[0x560c6094f9c1]
ffmpeg(+0xcd94)[0x560c60931d94]
/lib64/libc.so.6(__libc_start_main+0xfc)[0x7fea668537cc]
ffmpeg(+0xec69)[0x560c60933c69]
======= Memory map: ========

[a]

Replying to cehoyos:

where can I get Elephants_Dream_HD.avi , I try to reproduce this issue in master

[Carl Eugen Hoyos]

Not sure why you are asking me but here is a working link:
​https://www.mediaspip.net/IMG/avi/Elephants_Dream_HD.avi