rm: deadlock with fuzzed file

[tholin]

I found a deadlock in mpv with a fuzzed file. The problem appears to be in ffmpeg so I report it here directly.
Playing the file results in deadlock with rapid memory allocation and eventual oom.

$ gdb -args /home/cocobo/repository/mpv-build_fuzz/ffmpeg_build/ffmpeg -loglevel 99 -i rm_deadlock.rm 
GNU gdb (Gentoo 7.7.1 p1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/cocobo/repository/mpv-build_fuzz/ffmpeg_build/ffmpeg...done.
(gdb) r
Starting program: /home/cocobo/repository/mpv-build_fuzz/ffmpeg_build/ffmpeg -loglevel 99 -i rm_deadlock.rm
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-68139-g24e7e02 Copyright (c) 2000-2014 the FFmpeg developers
  built on Dec  2 2014 10:46:58 with gcc 4.8.3 (Gentoo 4.8.3 p1.1, pie-0.5.9)
  configuration: --prefix=/home/cocobo/repository/mpv-build_fuzz/build_libs --enable-static --disable-shared --enable-gpl --enable-avresample --enable-debug=gdb --disable-doc --disable-optimizations --disable-stripping
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 13.100 / 56. 13.100
  libavformat    56. 15.101 / 56. 15.101
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  2.103 /  5.  2.103
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'rm_deadlock.rm'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument 99.
Successfully parsed a group of options.
Parsing a group of options: input file rm_deadlock.rm.
Successfully parsed a group of options.
Opening an input file: rm_deadlock.rm.
[rm @ 0x1e4ab60] Format rm probed with size=2048 and score=100

Program received signal SIGINT, Interrupt.
0x00000000005923b1 in avio_r8 (s=0x1e4a360)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/aviobuf.c:510
510         if (s->buf_ptr >= s->buf_end)
(gdb) bt full
#0  0x00000000005923b1 in avio_r8 (s=0x1e4a360)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/aviobuf.c:510
No locals.
#1  0x00000000005928e7 in avio_rb16 (s=0x1e4a360)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/aviobuf.c:649
        val = 11890
#2  0x000000000059295d in avio_rb32 (s=0x1e4a360)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/aviobuf.c:665
        val = 779223040
#3  0x000000000066b3a4 in rm_read_audio_stream_info (s=0x1e4ab60, pb=0x1e4a360, st=0xaa60c480, 
    ast=0xaa60cd20, read_all=0) at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/rmdec.c:159
        flavor = 9
        sub_packet_h = 10
        sub_packet_size = 58
        bytes_per_minute = 149872
        coded_framesize = 580
        codecdata_length = 8
        buf = "\001\000\000\000\000\000\000\000\350\003\000\000\000\000\000\000\240\316\377\377\377\177\000\000\f\307\035\001\000\000\000\000\350\003\000\000\000\000\000\000\001\000\000\000\000\000\000\000@\317\377\377\377\177\000\000w\217\036\001\000\000\000\000\240\270\037\366\377\177\000\000\377\377\377\177", '\000' <repeats 12 times>, "\001\000\000\000\000\000\000\000t\317\377\377\377\177\000\000p\317\377\377\377\177", '\000' <repeats 22 times>, "\001\000\000\000\347\037g\365\377\177\000\000\001\000\000\000\350\003\000\000\000\000\000\000\000\000\000 \001\000\000\000\000\000\000\000\f\307\035\001", '\000' <repeats 11 times>, " `\243\344\001\000\000\000\000`\317\377\377\377\177\000\000"...
        version = 5
        ret = 0
#4  0x000000000066be97 in ff_rm_read_mdpr_codecdata (s=0x1e4ab60, pb=0x1e4a360, st=0xaa60c480, 
    rst=0xaa60cd20, codec_data_size=-128, mime=0x7fffffffd220 "audio7x-pn-realaudio")
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/rmdec.c:335
        v = 779248125
        size = 0
        codec_pos = 146
        ret = 0
#5  0x000000000066cc60 in rm_read_header (s=0x1e4ab60)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/rmdec.c:565
        rm = 0x1e4a1e0
        st = 0xaa60c480
        pb = 0x1e4a360
        tag = 1380992077
        tag_size = 164
        start_time = 0
        duration = 37150
        data_off = 394
        indx_off = 372439
        buf = "Audio Stream", '\000' <repeats 12 times>, "\370\257\344\001\000\000\000\000\320\061\345\001", '\000' <repeats 37 times>, "E\212\062\372-\005ː\322\377\377\377\177\000\000\274U]", '\000' <repeats 13 times>, "_\025h\365\377\177\000\000\000\000\000\000\000\000\000\000\b\323\377\377\377\177\000"
        mime = "audio7x-pn-realaudio\000\000\000\000`\243\344\001\000\000\000\000\210\322\377\377\n\000\000\000\000\000\000\000\377\177", '\000' <repeats 18 times>, "`\243\344\001\000\000\000\000`\243\344\001\000\000\000\000.RMF\000\000\000\022\000\001", '\000' <repeats 15 times>, "E\212\062\372-\005\313\300\322\377\377\377\177\000\000]V]\000\000\000\000"
        flags = 10
#6  0x00000000006c3f02 in avformat_open_input (ps=0x7fffffffd390, 
    filename=0x7fffffffde64 "rm_deadlock.rm", fmt=0x0, options=0x1e42498)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/libavformat/utils.c:463
        s = 0x1e4ab60
        ret = 100
        tmp = 0x1e49d40
        id3v2_extra_meta = 0x0
#7  0x0000000000410fab in open_input_file (o=0x7fffffffd470, filename=0x7fffffffde64 "rm_deadlock.rm")
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/ffmpeg_opt.c:873
        f = 0x0
        ic = 0x1e4ab60
        file_iformat = 0x0
        err = 0
        i = 48
        ret = 0
        timestamp = 17179869184
---Type <return> to continue, or q <return> to quit---
        opts = 0x1209477
        unused_opts = 0x0
        e = 0x0
        orig_nb_streams = 0
        video_codec_name = 0x0
        audio_codec_name = 0x0
        subtitle_codec_name = 0x0
        scan_all_pmts_set = 1
#8  0x00000000004190fb in open_files (l=0x1e3a0d8, inout=0x1209477 "input", 
    open_file=0x4108b3 <open_input_file>)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/ffmpeg_opt.c:2699
        g = 0x1e42470
        o = {g = 0x1e42470, start_time = -9223372036854775808, format = 0x0, codec_names = 0x0, 
          nb_codec_names = 0, audio_channels = 0x0, nb_audio_channels = 0, audio_sample_rate = 0x0, 
          nb_audio_sample_rate = 0, frame_rates = 0x0, nb_frame_rates = 0, frame_sizes = 0x0, 
          nb_frame_sizes = 0, frame_pix_fmts = 0x0, nb_frame_pix_fmts = 0, input_ts_offset = 0, 
          rate_emu = 0, accurate_seek = 1, ts_scale = 0x0, nb_ts_scale = 0, dump_attachment = 0x0, 
          nb_dump_attachment = 0, hwaccels = 0x0, nb_hwaccels = 0, hwaccel_devices = 0x0, 
          nb_hwaccel_devices = 0, stream_maps = 0x0, nb_stream_maps = 0, audio_channel_maps = 0x0, 
          nb_audio_channel_maps = 0, metadata_global_manual = 0, metadata_streams_manual = 0, 
          metadata_chapters_manual = 0, attachments = 0x0, nb_attachments = 0, 
          chapters_input_file = 2147483647, recording_time = 9223372036854775807, 
          stop_time = 9223372036854775807, limit_filesize = 18446744073709551615, mux_preload = 0, 
          mux_max_delay = 0.699999988, shortest = 0, video_disable = 0, audio_disable = 0, 
          subtitle_disable = 0, data_disable = 0, streamid_map = 0x0, nb_streamid_map = 0, 
          metadata = 0x0, nb_metadata = 0, max_frames = 0x0, nb_max_frames = 0, bitstream_filters = 0x0, 
          nb_bitstream_filters = 0, codec_tags = 0x0, nb_codec_tags = 0, sample_fmts = 0x0, 
          nb_sample_fmts = 0, qscale = 0x0, nb_qscale = 0, forced_key_frames = 0x0, 
          nb_forced_key_frames = 0, force_fps = 0x0, nb_force_fps = 0, frame_aspect_ratios = 0x0, 
          nb_frame_aspect_ratios = 0, rc_overrides = 0x0, nb_rc_overrides = 0, intra_matrices = 0x0, 
          nb_intra_matrices = 0, inter_matrices = 0x0, nb_inter_matrices = 0, 
          chroma_intra_matrices = 0x0, nb_chroma_intra_matrices = 0, top_field_first = 0x0, 
          nb_top_field_first = 0, metadata_map = 0x0, nb_metadata_map = 0, presets = 0x0, nb_presets = 0, 
          copy_initial_nonkeyframes = 0x0, nb_copy_initial_nonkeyframes = 0, copy_prior_start = 0x0, 
          nb_copy_prior_start = 0, filters = 0x0, nb_filters = 0, filter_scripts = 0x0, 
          nb_filter_scripts = 0, reinit_filters = 0x0, nb_reinit_filters = 0, fix_sub_duration = 0x0, 
          nb_fix_sub_duration = 0, canvas_sizes = 0x0, nb_canvas_sizes = 0, pass = 0x0, nb_pass = 0, 
          passlogfiles = 0x0, nb_passlogfiles = 0, guess_layout_max = 0x0, nb_guess_layout_max = 0, 
          apad = 0x0, nb_apad = 0, discard = 0x0, nb_discard = 0}
        i = 0
        ret = 0
#9  0x0000000000419288 in ffmpeg_parse_options (argc=5, argv=0x7fffffffda08)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/ffmpeg_opt.c:2736
        octx = {global_opts = {group_def = 0x12075f0 <global_group>, arg = 0x1205cdb "", 
            opts = 0x1e3a090, nb_opts = 1, codec_opts = 0x0, format_opts = 0x0, resample_opts = 0x0, 
            sws_opts = 0x0, swr_opts = 0x0}, groups = 0x1e3a0c0, nb_groups = 2, cur_group = {
            group_def = 0x0, arg = 0x0, opts = 0x0, nb_opts = 0, codec_opts = 0x0, format_opts = 0x0, 
            resample_opts = 0x0, sws_opts = 0x0, swr_opts = 0x0}}
        error = "\000\000\000\000\000\000\000\000\342b \001", '\000' <repeats 28 times>, "\346\am\000\000\000\000\000\360\330\377\377\377\177\000\000c\330A\000\000\000\000\000\342b \001\000\000\000\000\060\aL\001\001\000\000\000\000P\000\000\005\000\000\000\277\000\000\000\061\n\000\000\000\003\034\177\025\004\000\001\000\021\023\032\000\022\017\027\026", '\000' <repeats 14 times>
        ret = 0
#10 0x000000000042c7e7 in main (argc=5, argv=0x7fffffffda08)
    at /home/cocobo/repository/mpv-build_fuzz/ffmpeg/ffmpeg.c:3921
        ret = 32767
        ti = 0

analysis:

rm_read_header() in ffmpeg/libavformat/rmdec.c:488 loops over header tags until it reached the data section or eof.
The code for parsing MDPR tags tries to skip part of the codec info but treats the size of the codec block as signed.
A corrupt header can caused the skip to seek backwards.

int
ff_rm_read_mdpr_codecdata (AVFormatContext *s, AVIOContext *pb,
                           AVStream *st, RMStream *rst, int codec_data_size, const uint8_t *mime) rmdec.c:307
{

...

skip:
    /* skip codec info */
    size = avio_tell(pb) - codec_pos;
    avio_skip(pb, codec_data_size - size);  <--- codec_data_size = -128 

    return 0;
}

The seek ends up at the begginning of the same MDPR tags and rm_read_header() will try to decode the same MDPR tag indefinitely.
If the MDPR tag contains an audio stream rm_read_audio_stream_info() will continuously allocate memory which quickly results in oom.

[Carl Eugen Hoyos]

Regression since efb5fa79 / 139f3ac4

[Michael Niedermayer]

Fixed in a6f730730b82645a9d31aad0968487cb77d6946c