Stability issues with mpegts streams

[Thomas Hutschenreuther]

I have a scenario where I need to permanently decode mpegts-over-ip streams.
For stability testing purposes, I have set up a dvb-t receiver from which I forward 4 streams via vlc.
The content of the streams is as follows:

LD_LIBRARY_PATH=./libavcodec:./libavdevice:./libavfilter:./libavformat:./libavresample:./libavutil:./libpostproc:./libswresample:./libswscale ./ffprobe -loglevel 40 udp://192.168.1.30:5555
ffprobe version N-42024-g4bfb2ef Copyright (c) 2007-2012 the FFmpeg developers
  built on Jun 29 2012 13:41:49 with gcc 4.6.3
  configuration: --disable-stripping --enable-debug --enable-shared --enable-pic --disable-optimizations --enable-ffplay
  libavutil      51. 63.100 / 51. 63.100
  libavcodec     54. 29.101 / 54. 29.101
  libavformat    54. 11.100 / 54. 11.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     3.  0.100 /  3.  0.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
[mpegts @ 0x17c6220] Unable to seek back to the start
[mpegts @ 0x17c6220] PES packet size mismatch
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x1814520] mpeg_decode_postinit() failure
[mpeg2video @ 0x182d6c0] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpegts @ 0x17c6220] PES packet size mismatch
[mpeg2video @ 0x1814520] mpeg_decode_postinit() failure
[mpegts @ 0x17c6220] PES packet size mismatch
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpegts @ 0x17c6220] PES packet size mismatch
    Last message repeated 1 times
[mpeg2video @ 0x182d6c0] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x1814520] Warning MVs not available
[mpeg2video @ 0x1814520] concealing 6 DC, 6 AC, 6 MV errors
[mpegts @ 0x17c6220] PES packet size mismatch
[mpeg2video @ 0x182d6c0] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x182d6c0] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x182d6c0] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpegts @ 0x17c6220] PES packet size mismatch
[mpeg2video @ 0x1820e40] mpeg_decode_postinit() failure
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
[mpeg2video @ 0x1820e40] ac-tex damaged at 22 25
[mpeg2video @ 0x1820e40] Warning MVs not available
[mpeg2video @ 0x1820e40] concealing 45 DC, 45 AC, 45 MV errors
[mpeg2video @ 0x17fa260] mpeg_decode_postinit() failure
    Last message repeated 1 times
[mpegts @ 0x17c6220] PES packet size mismatch
    Last message repeated 33 times
[mpegts @ 0x17c6220] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegts, from 'udp://192.168.1.30:5555':
  Duration: N/A, start: 31366.006967, bitrate: 60512 kb/s
  Program 1000 
    Metadata:
      service_name    : MDR Sachsen
      service_provider: mufintv
    Stream #0:0[0x3ea](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 128 kb/s
    Stream #0:1[0x3e9]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 15000 kb/s, 29.41 fps, 25 tbr, 90k tbn, 50 tbc
  Program 2000 
    Metadata:
      service_name    : rbb Brandenburg
      service_provider: mufintv
    Stream #0:2[0x7d2](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 128 kb/s
    Stream #0:3[0x7d1]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
  Program 3000 
    Metadata:
      service_name    : WDR Koeln
      service_provider: mufintv
    Stream #0:4[0xbba](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 128 kb/s
    Stream #0:5[0xbb9]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 15000 kb/s, 27.78 fps, 25 tbr, 90k tbn, 50 tbc
  Program 4000 
    Metadata:
      service_name    : Bayerisches FS Nord
      service_provider: mufintv
    Stream #0:6[0x44](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16, 128 kb/s
    Stream #0:7[0x45]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 15000 kb/s, 26.34 fps, 25 tbr, 90k tbn, 50 tbc

I decode the audio content of those streams using ffmpeg.

After some time, several erroneous behaviors can be observed.
The first one is the one reported in #1475.

Another one is a crash (segfault) in mp_decode_layer3 in mpegaudiodec.c. That is strange as such, since there is no mp3 content in my streams.
I cannot give more detailed information since even with optimizations turned off and debbuging symbols turned on, the topmost stackframe looks corrupted.
Inspecting the network input buffer in gdb at that point (the fifo in the UDPContext) shows that a correct PES packet containing an mp2 header would be only a few bytes away.

That lead to the conclusion that again, as in #1475, a wrong PES packet is read from the stream by the mpegts demuxer.

For me, all those problems could be solved by turning off the possibility of spawning new PES streams after the probing phase.
I did this by removing line 1954 in mpegts.c:
ts->auto_guess = 1.

This change is also motivated by the fact that for me it looks as if the input stream goes out of sync in read_packet() in mpegts.c, i.e. the first byte in the input buffer
is different from 0x47, then the next 0x47 found is taken as the beginning of the PES packet. Since this is only one byte, which may well appear within a valid
PES packet, the probability of reading a faulty packet and misinterpreting it with various unwanted results is quite high.
If a new PES stream is spawned because of such a misinterpretation, the effects described above will appear.

I have done several tests and the problematic behavior with auto-guessing turned on "reproducibly" started in a range of 30 minutes to one day after starting ffmpeg.

With auto-guessing turned off, my application using ffmpeg has now been running without problems for five days.
The memory size also is the same as on start of the application.

The described behavior occurs both with my application, which uses the ffmpeg lgpl libraries, and with the ffmpeg executable.

[Carl Eugen Hoyos]

Replying to mufin:

Another one is a crash (segfault) in mp_decode_layer3 in mpegaudiodec.c. That is strange as such, since there is no mp3 content in my streams.
I cannot give more detailed information since even with optimizations turned off and debbuging symbols turned on, the topmost stackframe looks corrupted.

Did you try static compilation with --disable-asm --disable-yasm --disable-optimizations --enable-debug=3 ?
An alternative is to use valgrind.

[Thomas Hutschenreuther]

Valgrind output

[Thomas Hutschenreuther]

Replying to cehoyos:

Replying to mufin:

Another one is a crash (segfault) in mp_decode_layer3 in mpegaudiodec.c. That is strange as such, since there is no mp3 content in my streams.
I cannot give more detailed information since even with optimizations turned off and debbuging symbols turned on, the topmost stackframe looks corrupted.

Did you try static compilation with --disable-asm --disable-yasm --disable-optimizations --enable-debug=3 ?
An alternative is to use valgrind.

I will try static the compilation, you suggested. The problem is, that the solution I proposed works for me and I have only a finite amount of time for further experiments (reproducing the faulty behavior always takes quite some time).

I already have tried valgrind, but appearently it slows down processing so much, that completely other effects occur and the problem I describe does not appear (at least not after 1.5 days).

I have seen some uninitialized jump or move warnings but those were in different code sections.
And valgrind somehow raises a SIGILL at some point in time, which I think comes from valgrind not knowing
a certain instruction. I have attached an example Valgrind output of our application.

[Michael Niedermayer]

Please upload a stream with which the problems can be reproduced. Having a off line stream will also solve the "valgrind is too slow issue"

[Carl Eugen Hoyos]

Please reopen if you can either provide a sample or the necessary gdb output.