A/V desync on transport stream

[Carl Eugen Hoyos]

​http://streams.videolan.org/issues/5203/ts720p_i_fr.ts

The stream contains 720p60 H264 video and mp2 stereo sound. When transcoded with FFmpeg, the resulting file is non-interleaved, i.e. the 15 seconds of sound come first (without picture). MPlayer's mpegts demuxer indicates that the stream does start with mostly identical timestamps for audio and video (but plays the video too fast).
FFmpeg seems to believe that audio starts with timestamps ~130 and video with ~260.

$ ffmpeg -i ts720p_i_fr.ts -qscale 5 out.avi
ffmpeg version N-32263-g3584b01, Copyright (c) 2000-2011 the FFmpeg developers
  built on Sep  1 2011 12:14:35 with gcc 4.5.3
  configuration: --cc='/usr/local/gcc-4.5.3/bin/gcc -m32'
  libavutil    51. 14. 0 / 51. 14. 0
  libavcodec   53. 12. 0 / 53. 12. 0
  libavformat  53. 10. 0 / 53. 10. 0
  libavdevice  53.  3. 0 / 53.  3. 0
  libavfilter   2. 37. 0 /  2. 37. 0
  libswscale    2.  0. 0 /  2.  0. 0
[mpegts @ 0x8c963a0] Continuity Check Failed
[mpegts @ 0x8c963a0] max_analyze_duration 5000000 reached at 5016000
[mpegts @ 0x8c963a0] PES packet size mismatch
Input #0, mpegts, from 'ts720p_i_fr.ts':
  Duration: 00:00:14.61, start: 130.272000, bitrate: 9237 kb/s
  Program 16
    Metadata:
      service_name    : PMR COBRA 1213
      service_provider: PMR
    Stream #0.0[0x109]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0.1[0x103]: Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16, 128 kb/s
[buffer @ 0x8d86a00] w:1280 h:720 pixfmt:yuv420p tb:1/1000000 sar:1/1 sws_param:
Incompatible sample format 's16' for codec 'ac3', auto-selecting format 'flt'
Output #0, avi, to 'out.avi':
  Metadata:
    ISFT            : Lavf53.10.0
    Stream #0.0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 tbn, 60 tbc
    Stream #0.1: Audio: ac3 ([0] [0][0] / 0x2000), 48000 Hz, stereo, flt, 128 kb/s
Stream mapping:
  Stream #0.0 -> #0.0 (h264 -> mpeg4)
  Stream #0.1 -> #0.1 (mp2 -> ac3)
Press [q] to stop, [?] for help
[mpegts @ 0x8c963a0] Continuity Check Failed
    Last message repeated 1 times   236kB time=00:00:14.27 bitrate= 135.6kbits/s
[mpegts @ 0x8c963a0] PES packet size mismatch
[h264 @ 0x8c9b6e0] error while decoding MB 76 41, bytestream (-5)
[h264 @ 0x8c9b6e0] concealing 293 DC, 293 AC, 293 MV errors
frame=  852 fps= 31 q=5.0 Lsize=  177161kB time=00:00:14.62 bitrate=99241.4kbits/s
video:12002kB audio:228kB global headers:0kB muxing overhead 1348.534109%

[wg]

Replying to cehoyos:

The stream contains 720p60 H264 video and mp2 stereo sound. When transcoded with FFmpeg, the resulting file is non-interleaved, i.e. the 15 seconds of sound come first (without picture). MPlayer's mpegts demuxer indicates that the stream does start with mostly identical timestamps for audio and video (but plays the video too fast).
FFmpeg seems to believe that audio starts with timestamps ~130 and video with ~260.

ffprobe shows this file to have no video timestamps at all, so what should ffmpeg do except waiting for such a timestamp?
Please try

ffmpeg_g -i ts720p_i_fr.ts -qscale 5 out.avi -map 0.0:0.1 -map 0.1

maybe it does more like what you want. I could hear no sound neither in the original nor in the transcoded file.

[Carl Eugen Hoyos]

The sample does contain 14.6 seconds of sound. This is nearly exactly the intended video playtime.

[Mike]

Hi Carl. The bitstream looks good to me. The transport stream DTS/PTS are all good. The video has no B frames (I,P only), and uses PTS==DTS (which is legal).

This appears to be the same bug I already fixed in trac #88. My own media player, which includes the trac #88 patch, plays this stream fine. I hear audio immediately at the start.

I don't have time to look at latest git today, but my guess is that compute_pkt_fields() is corrupting the perfectly-good DTS/PTS timestamps passed to it.

Mike

P.S. the audio volume is really, really low in this clip. Peaks are at only 5% of full-scale.

[Mike]

I looked at this sample running the latest videolan git code, downloaded 2011-09-10. For me, ffprobe does show dts/pts timestamps in the file, but the PTS timestamps shown by ffprobe are wrong. The PTS values displayed are multiplied by approximately 2.

I confirmed this pts corruption is being done by compute_pkt_fields(), as my previous comment guessed. The line that changes pts to the wrong value is line 1044 in libavformat/utils.c:

pkt->pts = pkt->dts + pc->pts_dts_delta * num / den;

No time to debug this further right now.

[Carl Eugen Hoyos]

I can confirm that changing that line into "pkt->pts = pkt->dts" fixes transoding / playing this sample.

Forcing low_delay (and your patch from ticket #88) does not help, so this seems not to be the reason for the problem.

[Mike]

I analyzed this file in greater depth. The ffmpeg h264 decoder is doing the right thing, it is the bitstream that is wrong. I see two problems, which are completely unrelated.

1. The video stream was re-muxed at the the wrong frame rate. The original h264 elementary stream is 60.0 frames per second, but the .ts container was created using 30.0 frames per second.

2. This h264 elementary stream contains "buffering period SEI" and "pic timing SEI". The value in the pic timing dpb_output_delay field is completely wrong. It is typically a small integer value like "2", but in this sample it is 15628 (and it increments by 2 for each frame.) I'd guess this is a firmware error in the camcorder that created the compressed video stream.

These SEI are optional in h264, and they are not needed for this stream. So, a workaround for this camcorder bug would be to re-mux the stream and tell the re-muxer to remove all the pic timing SEI NALs.

I don't see an easy way for the h264 decoder to work around this bug. The way I read the h.264 spec, such a large value for dpb_output_delay is legal, but it is very, very unusual.

[Carl Eugen Hoyos]

The stream plays fine with WMP 12 (and mplayer -demuxer mpegts -mc 100), so I believe decoding should be improved.

[Carl Eugen Hoyos]

Ticket #1715 may be related.

[Carl Eugen Hoyos]

Possibly related to ticket #1866.

[Carl Eugen Hoyos]

Fixed by Michael in 1e5271a9fd6ddcceb083f2185a4bbd8d44c9a813 (FFmpeg 2.1), see also ticket #2143.