`-vsync cfr` tends to duplicate the wrong frames

[gdgsdg123]

With a VFR input, -vsync cfr may alter the playback experience by duplicating the wrong frames. (appeared to be... exactly 1 frame later)


Simple illustration of the problem:
(each number represents a frame of unique content)

-vsync vfr (expressed in CFR manner):

1 1 2 2 3

-vsync cfr:

1 2 2 3 3

Note:
The above happened in MP4, MKV container, failed to reproduce (seemed working correctly) in AVI container. (tested x264, and appeared to be codec dependent... as with magicyuv it happens in AVI also, and more severe: broken frames ^{(part of the duplicated frames have incorrect color)})

[gdgsdg123]

Replying to gdgsdg123:

...failed to reproduce (seemed working correctly) in AVI container.

Managed to reproduce it in AVI container:

ffmpeg -i "123.avi" -vsync cfr "cfr.avi"

Frame information of ​123.avi:

Frame Number	Presentation Timestamp	Duration	Content
1	0	2	1
2	2	2	2
3	4	N/A	3

Expected frame information of cfr.avi:

Frame Number	Presentation Timestamp	Duration	Content
1	0	1	1
2	1	1	2
3	2	1	2
4	3	1	3
5	4	N/A	3

[Gyan]

vsync takes effect before encoding and muxing so choice of encoder and muxer is irrelevant - one caveat being that if vsync is not specified by the user ffmpeg will set it as per muxer flags.

[gdgsdg123]

Replying to Gyan:

one caveat being that if vsync is not specified by the user ffmpeg will set it as per muxer flags.

In another word: always explicitly set the preferred vsync method, unless you want FFmpeg to choose one randomly for you...

[Gyan]

It's not random. It's chosen as per muxer flags which are fixed in code and rarely changed.

[gdgsdg123]

But random enough for unconscious users to be considered random...

[Carl Eugen Hoyos]

While the description is highly misleading, I believe that a (minor) issue can be reproduced with the sample file.

[gdgsdg123]

Replying to cehoyos:

...the description is highly misleading.

How?..


Replying to cehoyos:

...a (minor) issue can be reproduced with the sample file.

The issue is first observed in (and experimented with) production clips.

I specifically made the "123.avi" to demonstrate things better.

[Carl Eugen Hoyos]

Replying to gdgsdg123:

Replying to cehoyos:

...a (minor) issue can be reproduced with the sample file.

The issue is first observed in (and experimented with) production clips.

Feel free to provide such clips.

[gdgsdg123]

Replying to cehoyos:

Feel free to provide such clips.

I have already... If you had the interest to download:
​https://www.youtube.com/playlist?list=PLXvoB-d-4W1kKvr3CA20DrcU1HzTXibAX (check the descriptions)

[James Johnston]

I also encountered this issue. I've attached a small 6 frame test case named Null-Frame-Example.avi. I've also attached screenshots of how the null frame is supposed to look, and how it looks after FFmpeg has incorrectly handled it.

This test case is video that was initially recorded from a capture device using VirtualDub 1.9, and then trimmed using VirtualDub2 using Direct Stream Copy mode (i.e. no recompression).

During capture, VirtualDub inserted what are apparently known as "null frames" or "drop frames" in the AVI file.

Here is a table of frames in the attached test case.

Frame number	Frame type
0	Key frame
1	Key frame
2	Key frame
3	Null/drop frame
4	Key frame
5	Key frame

If I visually inspect the input file in VirtualDub2, 6 total frames are shown. Viewing frame index 3 will show the frame data from frame 2, and is the output I would also expect from FFmpeg. It is denoted with the special [D] tag in the status bar, indicating it is a null/drop frame. That is, a null frame is effectively a way of repeating the previous frame. And, that is how VirtualDub inserts placeholder frames during capture when the frames from the capture device are not keeping up with the clock.

In other words, this is the expected visual output of FFmpeg, and the observed frame table in VirtualDub2 when viewing the input:

Frame number	Frame data
0	Unique
1	Unique
2	Unique
3	Duplicate of frame 2
4	Unique
5	Unique

Now, suppose I run the following command:

# NOTE: using "-vsync cfr" will yield the same result
ffmpeg -i Null-Frame-Example.avi -fps_mode cfr -c:a flac -c:v ffv1 Null-Frame-Example.mkv

I then open this output MKV file, and observe the following actual / erroneous output:

Frame number	Frame data
0	Unique
1	Unique
2	Unique
3	Duplicate of frame 4
4	Unique
5	Unique

This is not correct; the frame index 3 should be from frame 2, as seen in the earlier table.

(Side note: I did not know that AVI files could even have these kinds of frames (most end users probably don't). My first attempt without any "vsync" or "fps_mode" parameter at all led to totally broken frames in the output file: the MKV file still has 6 frames according to VirtualDub2, but it gives a "frame missing" error when trying to view frame 3. It might be worth a consideration to make the default settings do the right thing somehow / always duplicate the (correct) frame when encountering null/drop frames.)

(How I got here: AviSynth's FFmpeg-based sources would silently drop these null frames, leading to audio/video sync issues in the output file!)

[MasterQuestionable]

͏    ​https://trac.ffmpeg.org/raw-attachment/ticket/8063/123.avi
͏    (~ 24.09 KiB; AVI: H.264 (AVC) video, 5 s, 1 FPS VFR (avg: 0.6, min: 0.5, max: 1), 800x600, RGB, ~ 18.42 KiB; ... ~ 5.67 KiB)

͏    ffprobe -v warning -hide_banner -threads 0 -show_entries "frame" -select_streams v:0 -of "xml" "123.avi" -o "123.avi.xml"

͏    Note:
͏    The interpreted "duration" alike appeared wrong.
͏    Would it be the cause? ("-vsync" misused which?)

[MasterQuestionable]

͏    Rearranged image:

͏    I believe ͏"123.avi" should be the most straight-forward sample.

͏    See [ ​https://trac.ffmpeg.org/ticket/10928 ] for the Trac bug...

[MasterQuestionable]

͏    Seemingly container dependent... probably for alike "hidden secrets" in comment:3?
͏    See also: https://ffmpeg.org/ffmpeg.html#Advanced-options

͏    "and more severe: broken frames (part of the duplicated frames have incorrect color)"
͏    Looks like bad Inter-compression handling. Codec bug maybe?
͏    Or wrong frame sent to codec per the problem..?

͏    Alike 1-frame-off issues may be serious:
͏    ​https://trac.ffmpeg.org/ticket/11060