ffmpeg process hangs with 100% CPU when getting a live rtsp stream (on redhat linux)

[muthu]

Summary of the bug:
ffmpeg process hangs with 100% CPU when getting a live rtsp stream (on redhat linux). This happens when 25 or more such processes are concurrently invoked.
How to reproduce:
On redhat linux (2.6+) launch 25 or more parallel live rtsp stream reads.

Command line is as follows,
/home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov

The generated console (stats) data is,
ffmpeg version 1.0 Copyright (c) 2000-2012 the FFmpeg developers

built on Nov 12 2012 13:10:29 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-44)
configuration: --disable-yasm --enable-nonfree --prefix=/home/lsadm/StressTest/FFmpeg/
libavutil 51. 73.101 / 51. 73.101
libavcodec 54. 59.100 / 54. 59.100
libavformat 54. 29.104 / 54. 29.104
libavdevice 54. 2.101 / 54. 2.101
libavfilter 3. 17.100 / 3. 17.100
libswscale 2. 1.101 / 2. 1.101
libswresample 0. 15.100 / 0. 15.100

[h264 @ 0x17d6f540] Missing reference picture
[h264 @ 0x17d6f540] decode_slice_header error
[h264 @ 0x17d6f540] concealing 300 DC, 300 AC, 300 MV errors in P frame
[h264 @ 0x17d6f540] Missing reference picture
[h264 @ 0x17d6f540] RTP: missed 15429 packets
[h264 @ 0x17d6f540] RTP: missed 15430 packets
[h264 @ 0x17d6f540] RTP: missed 15431 packets
[h264 @ 0x17d6f540] RTP: missed 15432 packets
[h264 @ 0x17d6f540] RTP: missed 15433 packets

Below is a snapshot of the CPU captured with top command,
Tasks: 404 total, 2 running, 402 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.4%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6096740k total, 5990044k used, 106696k free, 10404k buffers
Swap: 1020116k total, 336084k used, 684032k free, 3099316k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

22478 lsadm 25 0 27364 4732 1920 R 99.9 0.1 18:35.66 ffmpeg
16383 lsadm 15 0 2695m 2.3g 2624 S 0.7 40.4 32:52.16 perl
26991 lsadm 15 0 29340 6272 2776 S 0.3 0.1 0:00.73 ffmpeg
26994 lsadm 15 0 28192 4904 2416 S 0.3 0.1 0:00.78 ffmpeg
27319 lsadm 15 0 10988 1280 772 R 0.3 0.0 0:00.07 top

1 root 15 0 10344 80 48 S 0.0 0.0 3:56.92 init

[lsadm@grinder025 Scripts]$ ps -aef | grep 22478
lsadm 22478 22476 99 17:30 pts/1 00:28:54 /home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov
lsadm 29402 2890 0 17:59 pts/3 00:00:00 grep 22478
[lsadm@grinder025 Scripts]$ ps -aef | grep 22476
lsadm 22476 16383 0 17:30 pts/1 00:00:00 sh -c /home/lsadm/StressTest/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/5EErJWzqY7hSRkrUL030_A_QVGA+email.share@yahoo.com-5EErJWzqY7hSRkrUL030_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-97_5EErJWzqY7hSRkrUL030.log

[Carl Eugen Hoyos]

Please test current git head and provide a backtrace.

[muthu]

Replying to cehoyos:

Please test current git head and provide a backtrace.

Thanks for your quick response.

I downloaded the current version from (​http://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2) and still see the issue after some time (15 minutes or so).
Below are the details,

Command line is,
/home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log

The generated console (stats) data is,
[lsadm@grinder025 Scripts]$ cat /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log
ffmpeg version 1.0.git Copyright (c) 2000-2012 the FFmpeg developers

built on Nov 30 2012 12:44:30 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-44)
configuration: --prefix=/home/lsadm/test-muthu/newerffmpeg/FFmpeg/ --enable-gpl --enable-version3 --enable-nonfree --disable-yasm
libavutil 52. 9.102 / 52. 9.102
libavcodec 54. 77.100 / 54. 77.100
libavformat 54. 39.100 / 54. 39.100
libavdevice 54. 3.100 / 54. 3.100
libavfilter 3. 23.104 / 3. 23.104
libswscale 2. 1.103 / 2. 1.103
libswresample 0. 17.101 / 0. 17.101
libpostproc 52. 2.100 / 52. 2.100

[h264 @ 0x1b50e520] Missing reference picture, default is 0
[h264 @ 0x1b50e520] decode_slice_header error
[h264 @ 0x1b50e520] concealing 300 DC, 300 AC, 300 MV errors in P frame
[h264 @ 0x1b50e520] Missing reference picture, default is 14
[h264 @ 0x1b50e520] RTP: missed 7594 packets
[h264 @ 0x1b50e520] RTP: missed 7595 packets
[h264 @ 0x1b50e520] RTP: missed 7596 packets
[h264 @ 0x1b50e520] RTP: missed 7597 packets
[h264 @ 0x1b50e520] RTP: missed 7598 packets
[h264 @ 0x1b50e520] RTP: missed 7599 packets
[h264 @ 0x1b50e520] RTP: missed 7600 packets
[h264 @ 0x1b50e520] RTP: missed 7601 packets
[h264 @ 0x1b50e520] RTP: missed 7602 packets
[lsadm@grinder025 Scripts]$

Below is a snapshot of the CPU captured with top command,
top - 13:37:40 up 52 days, 2:06, 10 users, load average: 3.51, 5.50, 4.97
Tasks: 473 total, 3 running, 470 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 0.3%sy, 0.0%ni, 96.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6096740k total, 6073196k used, 23544k free, 8528k buffers
Swap: 1020116k total, 361720k used, 658396k free, 2466924k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

18206 lsadm 25 0 27792 4808 1972 R 99.9 0.1 18:29.11 ffmpeg
22322 lsadm 25 0 27968 4992 1976 R 99.9 0.1 6:42.93 ffmpeg
13406 lsadm 15 0 3933m 2.9g 2628 S 3.9 49.3 29:05.40 perl
23117 lsadm 15 0 28564 4976 2484 S 2.0 0.1 0:00.61 ffmpeg
23311 lsadm 15 0 28556 4964 2484 S 2.0 0.1 0:00.49 ffmpeg
23768 lsadm 15 0 11116 1232 684 R 2.0 0.0 0:00.01 top

1 root 15 0 10344 80 48 S 0.0 0.0 3:56.94 init

[lsadm@grinder025 Scripts]$ ps -aef | grep 18206
lsadm 18206 18205 99 13:19 pts/1 00:19:31 /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov
lsadm 23831 13531 0 13:38 pts/7 00:00:00 grep 18206

[lsadm@grinder025 Scripts]$ ps -aef | grep 18205
lsadm 18205 13406 0 13:19 pts/1 00:00:00 sh -c /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i 'rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp' -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov 2> /home/lsadm/StressTest/Outputs/openRTSPLog/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.log
lsadm 18206 18205 99 13:19 pts/1 00:19:29 /home/lsadm/test-muthu/newerffmpeg/FFmpeg/bin/ffmpeg -i rtsp://xstream-bcast-grinder003:50554/sEOrJIzkY7U1S7OYLaNL_A_QVGA+email.share@yahoo.com-sEOrJIzkY7U1S7OYLaNL_A_QVGA.sdp -timelimit 306 -minrate 70k -maxrate 1400k -map 0 -t 306 -c:v copy -c:a alac -y /home/lsadm/StressTest/Outputs/mediaOutput/RTSP_DownStreamer-65_sEOrJIzkY7U1S7OYLaNL.mov
lsadm 23799 13531 0 13:38 pts/7 00:00:00 grep 18205
[lsadm@grinder025 Scripts]$

[Carl Eugen Hoyos]

If you report that FFmpeg hangs, I believe you, no need to post ps output.

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

[muthu]

Replying to cehoyos:

If you report that FFmpeg hangs, I believe you, no need to post ps output.

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

Thanks you.

I actually run/launch these from a perl script (launches 25 instances every ~6 minutes) and they fail at random (after 15-20 minutes or so). I don't think launching each invocation in gdb is feasible. I realize however that i can attach to an ffmpeg process & take a backtrace (cmd-line attach). I hope this is something that will work. Please correct me or point me in the right direction if this understanding is incorrect. I must admit i have been working in Java & am not too familiar with using gdb. So please forgive my amateurism.

Also please let me know how to build ffmpeg with debug enabled.

Thanks
Muthu

[Carl Eugen Hoyos]

Replying to muthuk:

Replying to cehoyos:

Since no developer is able to reproduce the problem (because of your rtsp address), a backtrace is necessary.
Please run "$ gdb ffmpeg_g", then "r -i rtsp... " and when ffmpeg hangs, you interrupt it with Ctrl-C and type "bt" to get a backtrace, please post it here.

I actually run/launch these from a perl script (launches 25 instances every ~6 minutes) and they fail at random (after 15-20 minutes or so). I don't think launching each invocation in gdb is feasible. I realize however that i can attach to an ffmpeg process & take a backtrace (cmd-line attach).

I have never done it myself but this sounds like a good idea.

Also please let me know how to build ffmpeg with debug enabled.

$ ./configure && make ;-)
Your configure line produces two executables, ffmpeg_g contains debug symbols, I suspect --disable-stripping leaves the symbols also in ffmpeg.
(I only realize now you are using --disable-yasm. This is actually a good idea for debugging, but don't forget to remove it in the future, it hurts performance.)

[Carl Eugen Hoyos]

As said, this sounds like an important issue, please reopen if you can provide a rtsp link that allows to reproduce the problem or a backtrace etc. as explained on ​http://ffmpeg.org/bugreports.html