High memory allocation with broken pmp file

[Carl Eugen Hoyos]

(vlc ticket 8226)
A sample was uploaded to videolan trac that allocates several Gigabyte at probing, I will attach the sample and a patch that improves the situation, but valgrind still reports 1,021,771,097 allocs with 1,902,250,215,888,401,222 bytes with the patch.

$ ffmpeg -i dos.pmp
ffmpeg version N-50197-g23c9180 Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb 23 2013 13:10:17 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack
  libavutil      52. 17.103 / 52. 17.103
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.100 / 54. 63.100
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 39.101 /  3. 39.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[pmp @ 0x32a8520] Unsupported audio format
[NULL @ 0x32ae8e0] [IMGUTILS @ 0x7fffe55dd7e0] Picture size 1094795585x1094795585 is invalid
[NULL @ 0x32ae8e0] Ignoring invalid width/height values
[pmp @ 0x32a8520] Could not find codec parameters for stream 0 (Video: mpeg4, yuv420p): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 1 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 2 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 3 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 4 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 5 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 6 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 7 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 8 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 9 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 10 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options

...

[pmp @ 0x32a8520] Could not find codec parameters for stream 16691 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16692 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16693 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16694 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16695 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16696 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16697 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16698 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16699 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16700 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16701 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16702 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16703 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16704 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[pmp @ 0x32a8520] Could not find codec parameters for stream 16705 (Audio: none, 1094795585 Hz, 4276546 channels): unspecified sample format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
dos.pmp: could not find codec parameters

[Elon Musk]

It should just stop when unsupported audio format is found. (It does not set codec id and because of that nothing is lost)

[Elon Musk]

If format really allow so many number of streams (16bit), it could be also sensible to demux only some of them (because of brain dead format), but considering there is no such real sample there is no need for such feature. And you could just safely limit max number of streams to 16.

[Hendrik]

Even if the file has 16k streams, why does that require gigabytes of memory?

[Elon Musk]

By calling avformat_new_stream() 16k times.

[Hendrik]

Is the stream structure so big that this adds up?

[Elon Musk]

Not just stream structure it allocates other stuff too.

[reimar]

You're looking at the completely wrong place, 16k streams is not a problem, and 16 streams is more than ridulously low, there are more than enough real-world files using more than 30 stream.
I'm quite sure the issue is in a missing EOF check.
However in addition the valgrind numbers you quote give a completely wrong impression, it is not possible to allocate than much data, this is just a huge number of reallocs.
I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error, if a user does not want large allocations they should forbid them (though I admit there might be a good argument to be made to default to a lower value on 32 bit systems).
The CPU usage is a different issue, and that is due to the second loop, which in the worst case iterates 2 billion times.

[Carl Eugen Hoyos]

Replying to reimar:

I do not think the allocation ever becomes unreasonably large, however we do have a av_max_alloc function and I have a suspicion this might be a user error

Allocation is >2G here.

[DonMoir]

It uses approximately 400MB opening 16706 streams most likely all for nothing. I have to assume that number is wrong and index_cnt is definitely out of the park at 1094778880.

The remaining allocations come from the second loop or at least more of it. I did not wait 1 billion times to find out :)

Does anyone have a sample pmp that works correctly?

I had to reboot after my first attempt at opening dos.pmp.

[Carl Eugen Hoyos]

Fixed by Reimar and Michael.