High memory consumption on remuxing a program stream to matroska

[Carl Eugen Hoyos]

​http://thread.gmane.org/gmane.comp.video.ffmpeg.user/49668
A user uploaded a program stream with a timestamp discontinuity to show very high memory usage on remuxing.
The original sample is 2G and uses ~600M on remuxing, I cut the sample to 300M, -benchmark reports a memory usage >200M here (600M memory consumption are needed for a 800M cut of the original sample). Remuxing fails without -fflags +genpts

$ ffmpeg -fflags +genpts -benchmark -i robbins-2_cut.vob -vcodec copy -sn -an out.mkv
ffmpeg version N-59860-g53e6977 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jan 15 2014 10:01:34 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 62.100 / 52. 62.100
  libavcodec     55. 48.101 / 55. 48.101
  libavformat    55. 23.103 / 55. 23.103
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  1.100 /  4.  1.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[mpeg2video @ 0x1ca4de0] Invalid frame dimensions 0x0.
Input #0, mpeg, from 'robbins-2_cut.vob':
  Duration: 26:10:39.22, start: 1543.086867, bitrate: 26 kb/s
    Stream #0:0[0x81]: Audio: ac3, 48000 Hz, mono, fltp, 192 kb/s
    Stream #0:1[0x80]: Audio: ac3, 48000 Hz, 5.1(side), fltp, 384 kb/s
    Stream #0:2[0x1e0]: Video: mpeg2video (Main), yuv420p(tv), 720x480 [SAR 32:27 DAR 16:9], max. 7500 kb/s, 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0:3[0x84]: Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:4[0x82]: Audio: ac3, 48000 Hz, mono, fltp, 192 kb/s
    Stream #0:5[0x83]: Audio: ac3, 48000 Hz, mono, fltp, 192 kb/s
    Stream #0:6[0x25]: Subtitle: dvd_subtitle
    Stream #0:7[0x24]: Subtitle: dvd_subtitle
    Stream #0:8[0x23]: Subtitle: dvd_subtitle
    Stream #0:9[0x22]: Subtitle: dvd_subtitle
    Stream #0:10[0x21]: Subtitle: dvd_subtitle
    Stream #0:11[0x20]: Subtitle: dvd_subtitle
    Stream #0:12[0x27]: Subtitle: dvd_subtitle
    Stream #0:13[0x26]: Subtitle: dvd_subtitle
Output #0, matroska, to 'out.mkv':
  Metadata:
    encoder         : Lavf55.23.103
    Stream #0:0: Video: mpeg2video (mpg2 / 0x3267706D), yuv420p, 720x480 [SAR 32:27 DAR 16:9], q=2-31, max. 7500 kb/s, 29.97 fps, 1k tbn, 90k tbc
Stream mapping:
  Stream #0:2 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 9455 fps=0.0 q=-1.0 Lsize=  245342kB time=00:06:34.16 bitrate=5099.1kbits/s
video:245240kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.041788%
bench: utime=0.616s
bench: maxrss=224672kB

[Carl Eugen Hoyos]

Cut sample uploaded to ​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket3308/

[Carl Eugen Hoyos]

Massif output for the remuxing of the complete file:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 77 19,609,853,744      647,132,288      643,912,769     3,219,519            0
99.50% (643,912,769B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->98.17% (635,266,133B) 0xCEE728: av_buffer_realloc (buffer.c:164)
| ->98.17% (635,264,092B) 0x61FC01: av_dup_packet (avpacket.c:204)
| | ->98.17% (635,264,092B) 0x5DF62D: parse_packet (utils.c:1286)
| | | ->98.17% (635,264,092B) 0x5E0036: read_frame_internal (utils.c:1397)
| | |   ->98.17% (635,264,092B) 0x5E0C10: av_read_frame (utils.c:1504)
| | |   | ->98.17% (635,264,092B) 0x47E482: process_input (ffmpeg.c:3031)
| | |   |   ->98.17% (635,264,092B) 0x4668DE: main (ffmpeg.c:3334)
| | |   |
| | |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.00% (2,041B) in 1+ places, all below ms_print's threshold (01.00%)
|
->01.34% (8,646,636B) in 19 places, all below massif's threshold (01.00%)