First frames incorrect after seeking on H264 file

[jrummell]

Summary of the bug:
While playing a H264 video, seeking to a different position appears to display the wrong frames. After a few (1-4?), subsequent frames look correct.

Input: ​https://storage.googleapis.com/chromiumos-test-assets-public/Shaka-Dash/480.mp4

How to reproduce:

ffplay 480.mp4

ffplay version N-86098-g3fefaea Copyright (c) 2003-2017 the FFmpeg developers

built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
configuration: --pkg-config-flags=--static --extra --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree
libavutil 55. 63.100 / 55. 63.100
libavcodec 57. 96.101 / 57. 96.101
libavformat 57. 72.101 / 57. 72.101
libavdevice 57. 7.100 / 57. 7.100
libavfilter 6. 89.101 / 6. 89.101
libswscale 4. 7.101 / 4. 7.101
libswresample 2. 8.100 / 2. 8.100
libpostproc 54. 6.100 / 54. 6.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '480.mp4':

Metadata:

major_brand : dash
minor_version : 0
compatible_brands: iso6mp41avc1
creation_time : 2015-09-01T00:38:39.000000Z

Duration: 00:07:10.00, start: 0.033367, bitrate: 626 kb/s

Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 854x480 [SAR 1:1 DAR 427:240], 15 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
Metadata:

creation_time : 2015-09-01T00:38:39.000000Z
handler_name : VideoHandler

Once playing, right click at various different points. In some places it's obvious that a different frame is displayed.

Example: seek to 46%. Person walking across the screen jumps several times before walking normally. Even if the video is paused when you seek, then when play is started again the same effect is seen.

Reference: ​http://crbug.com/568336

After playing with this for a while, it appears to be related to commit 4ab56667594842283dc5ae07f0daba2a2cb4d3af. My guess is that when seeking the first frame displayed is the one at the start of the fragment, but then the code catches up to the correct position quickly. Removing the call to either mov_read_sidx() or mov_seek_fragment() in libavformat/mov.c "fixes" the problem.

In Chromium, the effect is different. It gets the following frames out of order, so the video jumps around quite a bit. Observation video attached to the crbug listed above, if you're interested.

[Carl Eugen Hoyos]

As reported here, if there is an issue, it is a regression since 3166a6fc379789b3782f431dd232033c2069c443 (when mov_read_sidx() did not exist yet).
I don't know if this helps understanding the actual original issue better.
(Playback speed after seeking with FFplay in a video-only stream is not strictly defined.)

Or is this a duplicate of ticket #5090?

[Dale Curtis]

I don't think this is a duplicate of #5090. The video plays fine until you start seeking around in it a few times. In Chrome it takes ~2 seeks before you start getting repeated frames.

[Dale Curtis]

Updated my reply on the patch that caused this issue, ​http://ffmpeg.org/pipermail/ffmpeg-devel/2017-July/213025.html, with a repro using ffplay:

Just seeking a few times in ffplay can trigger this issue with the clip linked in my original message:

​http://storage.googleapis.com/dalecurtis-shared/buck480p30_na.mp4

./ffplay -v debug -drp 1 ~/Downloads/buck480p30_na.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination14583000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination14586000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] found fragment index for track 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] found fragment index entry for track 1 and moof_offset 16686198
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] found frag time 14589000, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14607000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14610000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14622000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14631000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14634000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbce00008c0] invalid dts/pts combination 14643000

Disabled sidx processing resolves this issue:

diff --git a/libavformat/mov.c b/libavformat/mov.c
index 63f84be782..919475f12f 100644
--- a/libavformat/mov.c
+++ b/libavformat/mov.c
@@ -5497,7 +5497,7 @@ static const MOVParseTableEntry
mov_default_parse_table[] = {

{ MKTAG('a','l','a','c'), mov_read_alac }, /* alac specific atom */
{ MKTAG('a','v','c','C'), mov_read_glbl },
{ MKTAG('p','a','s','p'), mov_read_pasp },

-{ MKTAG('s','i','d','x'), mov_read_sidx },
+ { MKTAG('s','i','d','x'), mov_read_sidx },

[Dale Curtis]

cc:rodger since he added this code.

[Dale Curtis]

The issue seems to be caused by the fact that after a couple seeks the mov demuxer starts vending pts timestamps in order instead of accounting for decode reordering. E.g.:

Good w/ and w/o sidx:
1st Seek : 50000
EnqueuePacket : pts: 4327323(48081.4), dts: 4324320(48.048 s)
EnqueuePacket : pts: 4330326(48114.7), dts: 4327323(48.0814 s)
EnqueuePacket : pts: 4339335(48214.8), dts: 4330326(48.1147 s)
EnqueuePacket : pts: 4333329(48148.1), dts: 4333329(48.1481 s)
EnqueuePacket : pts: 4336332(48181.5), dts: 4336332(48.1815 s)

Good w/o sidx:
2nd Seek : 15000
EnqueuePacket : pts: 963963(10710.7), dts: 960960(10.6773 s)
EnqueuePacket : pts: 972972(10810.8), dts: 963963(10.7107 s)
EnqueuePacket : pts: 966966(10744.1), dts: 966966(10.7441 s)
EnqueuePacket : pts: 969969(10777.4), dts: 969969(10.7774 s)
EnqueuePacket : pts: 975975(10844.2), dts: 972972(10.8108 s)

Bad w/ sidx:
2nd Seek : 15000
EnqueuePacket : pts: 963963(10710.7), dts: 960960(10.6773 s)
EnqueuePacket : pts: 966966(10744.1), dts: 963963(10.7107 s)
EnqueuePacket : pts: 969969(10777.4), dts: 966966(10.7441 s)
EnqueuePacket : pts: 972972(10810.8), dts: 969969(10.7774 s)
EnqueuePacket : pts: 975975(10844.2), dts: 972972(10.8108 s)

Note how the second set of pts values are now in order and thus no longer correct. I'm still digging to see what's going on here; suggestions welcome since I'm unfamiliar with this code. cc:michael in case he has any hints since rodger seems unresponsive.

[Dale Curtis]

Cause of bad pts seems to be incorrect duration data:

mov.c:6397 pkt->pts = pkt->dts + sc->dts_shift + sc->ctts_data[sc->ctts_index].duration;

A good playback has the following values:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f844b807200] pts calc dts:960960, dts_shift:0, duration:3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f844b807200] pts calc dts:963963, dts_shift:0, duration:9009
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f844b807200] pts calc dts:966966, dts_shift:0, duration:0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f844b807200] pts calc dts:969969, dts_shift:0, duration:0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f844b807200] pts calc dts:972972, dts_shift:0, duration:3003

While a bad playback has:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7faea000ca00] pts calc dts:960960, dts_shift:0, duration:3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7faea000ca00] pts calc dts:963963, dts_shift:0, duration:3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7faea000ca00] pts calc dts:966966, dts_shift:0, duration:3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7faea000ca00] pts calc dts:969969, dts_shift:0, duration:3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7faea000ca00] pts calc dts:972972, dts_shift:0, duration:3003

Trying to see now why ctts_data has bad durations.

[Dale Curtis]

Seems the code may be reading the wrong trun box?

good:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc410020c00] mov_read_trun, ctts_index: 320, duration: 3003, sample_duration:3003, file_position:860478
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc410020c00] mov_read_trun, ctts_index: 321, duration: 9009, sample_duration:3003, file_position:860478
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc410020c00] mov_read_trun, ctts_index: 322, duration: 0, sample_duration:3003, file_position:860478
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc410020c00] mov_read_trun, ctts_index: 323, duration: 0, sample_duration:3003, file_position:860478
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc410020c00] mov_read_trun, ctts_index: 324, duration: 3003, sample_duration:3003, file_position:860478

bad:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc060836800] mov_read_trun, ctts_index: 320, duration: 3003, sample_duration:3003, file_position:3350481
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc060836800] mov_read_trun, ctts_index: 321, duration: 3003, sample_duration:3003, file_position:3350481
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc060836800] mov_read_trun, ctts_index: 322, duration: 3003, sample_duration:3003, file_position:3350481
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc060836800] mov_read_trun, ctts_index: 323, duration: 3003, sample_duration:3003, file_position:3350481
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc060836800] mov_read_trun, ctts_index: 324, duration: 3003, sample_duration:3003, file_position:3350481

[Dale Curtis]

My best guess so far is that the ctts index adjustment during mov_seek_stream() is incorrect when not all trun boxes have been read; which was the case prior to the sidx support.

good:
Seek 50000:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7fff05b600] adjusting ctts index... old: 21, new:1440, count:12887

bad
Seek 50000:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f8700837000] adjusting ctts index... old: 21, new:480, count:640

[Carl Eugen Hoyos]

Is this issue still reproducible with current FFmpeg git head?

[Dale Curtis]

Should be fixed with ​http://git.videolan.org/?p=ffmpeg.git;a=commit;h=37e8edc9f51545ad91cbdf7dbe796af93f011abe

[Carl Eugen Hoyos]

Please mention tickets in future commit messages and please close tickets once they are fixed.