Position of key frame incorrect

[David Johansen]

Summary of the bug:
How to reproduce:

% ffprobe -show_entries packet=pos,flags -of csv=p=0 -i kf_pos_test.mp4 | grep K
ffprobe version 6.1.1 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr --enable-shared --disable-static --disable-debug --disable-doc --disable-ffplay --enable-gpl --enable-libfdk_aac --enable-libx264 --enable-libx265 --enable-librsvg --enable-openssl --enable-nonfree --enable-libaom --extra-libs=-lpthread
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'kf_pos_test.mp4':
  Metadata:
    major_brand     : iso5
    minor_version   : 512
    compatible_brands: iso5iso6mp41
    encoder         : Lavf60.16.100
  Duration: 00:00:09.92, start: 0.224000, bitrate: 835 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv), 1280x720, 748 kb/s, 15.15 fps, 15.15 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
1158,K__
106174,K__
205264,K__
309370,K__
414778,K__
516205,K__
620919,K__
725663,K__
827272,K__
931036,K__

That last entry should be 930976 so it's 60 bytes later than it should be. What can I do to help diagnose the cause of this?

Here's a file to demonstrate the issue:
​https://drive.google.com/file/d/1kE9eNNYnBq_Lr-il2YWXGH4nOZLWs2Nx/view?usp=sharing

[MasterQuestionable]

͏    Alike:
͏    ​https://trac.ffmpeg.org/attachment/ticket/11052/sample.mp4.xml

͏    ​https://drive.usercontent.google.com/download?id=1kE9eNNYnBq_Lr-il2YWXGH4nOZLWs2Nx
͏    (~ 1,011.36 KiB; MP4)

[David Johansen]

Looks like this was flagged as similar because they both have a Google Drive link, because looking at the actual ticket, they are very different

[MasterQuestionable]

͏    Sorry for the confusion.
͏    The Google download link is interpreted from yours. (optimized delivery)

[MasterQuestionable]

͏    Remuxing ("-c copy") again to MP4 via FFmpeg, or to MKV:
͏    Would the problem still persist?

[David Johansen]

How would I verify the result? In this case, I know what the results should be because I catted the keyframes together into a single file, but if I remuxed then I wouldn't have that info

[Balling]

Use hex editor and find out what is the offset.

Ib sill do not get why does this matter. What is your need to know the perfect offset of last keyframe?

Does it happen on other keyframes not in the end (on other files)?

[MasterQuestionable]

͏    No wonder then...
͏    The MP4 format may be a bit more (or much more...) complicated than that: mere concatenation may not properly present.

͏    In particular, it has the concepts similar to Matroska's Cluster.

[MasterQuestionable]

͏    "why does this matter"
<^>    Theoretical completeness.
͏    And potential hidden problems.

͏    Significant difference, tends to be caused by insignificant things.

[mkver]

There is 0x00015827 at offset 931036. This corresponds to a NAL unit with size 88103 which fits nicely with the packet size 88107 as reported by ffprobe. There is 0x0000047C at offset 930976. This is already invalid with a NALU length size of two. With a NALU length size of three, this would be a NALU with nal_unit_type 28 -- one of the unspecified ones. The next NALU would then have a size of 0x340000, which is larger than the whole file. With a nal unit length size of four, the next NALU has a length of more than 2GiB.

To sum it up, your offset 930976 (for which you provided no real explanation) is just wrong.

[David Johansen]

Replying to Balling:

Use hex editor and find out what is the offset.

Ib sill do not get why does this matter. What is your need to know the perfect offset of last keyframe?

Does it happen on other keyframes not in the end (on other files)?

Yes, I will find an example where it happens in the middle of the file

Replying to MasterQuestionable:

͏    No wonder then...
͏    The MP4 format may be a bit more (or much more...) complicated than that: mere concatenation may not properly present.

͏    In particular, it has the concepts similar to Matroska's Cluster.

I misrepresented what's being done slightly. These are fragmented mp4 files that are being made by the HLS muxer that I'm then combining into a single file so they're simpler to store/serve, but I need to know the location of the keyframes to be able to provide the necessary EXT-X-BYTERANGE info

Replying to mkver:

There is 0x00015827 at offset 931036. This corresponds to a NAL unit with size 88103 which fits nicely with the packet size 88107 as reported by ffprobe. There is 0x0000047C at offset 930976. This is already invalid with a NALU length size of two. With a NALU length size of three, this would be a NALU with nal_unit_type 28 -- one of the unspecified ones. The next NALU would then have a size of 0x340000, which is larger than the whole file. With a nal unit length size of four, the next NALU has a length of more than 2GiB.

To sum it up, your offset 930976 (for which you provided no real explanation) is just wrong.

Using the values that I provided for EXT-X-BYTERANGE it will playback in Video.js and Safari, but using the values that ffprobe provides it stops, so that's my current criteria for deciding if it's "correct"

[MasterQuestionable]

͏    Mere concatenation of bunch of MP4 cannot end in valid MP4...
͏    You have some misunderstanding on "concatenation".

͏    See also:
͏    ​https://stackoverflow.com/questions/35177797/what-is-fragmented-mp4-fmp4-and-how-is-it-different-than-normal-mp4#35180327

[Balling]

Mere concatenation of bunch of MP4 cannot end in valid

Yet segments of fmp4 do end up as perfect and valid mp4.

[Balling]

but using the values that ffprobe provides it stops

Then this is a bug, as many of them in Video.js

[MasterQuestionable]

͏    Seriously..?
͏    Won't the container data clash and do garbage?

͏    ----

͏    Unsure if it's Video.js bug: as much is unknown about the actual file.
͏    See also: ​https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.2.2

͏    Probably didn't set EXT-X-BYTERANGE correctly..?

͏    Seemingly problem in the definition:
͏    "If o is not present, a previous Media Segment MUST appear in the Playlist file and MUST be a sub-range of the same media resource, or ..."
<^>    What must be a sub-range of?

[David Johansen]

Here's an example video that I used to demonstrate what I'm doing and that the expected values don't match up:
​https://drive.google.com/file/d/1WxOrSi-GNB45nLUUiR4PT7c4H2VurtKk/view?usp=sharing

Run this command to make a fragmented mp4 with each fragment having a single key frame in it:
ffmpeg -i input.mp4 -c:v libx264 -copyts -muxdelay 0 -force_key_frames source -hls_time 0.9 -hls_init_time 0.9 -hls_list_size 0 -hls_flags independent_segments -hls_segment_type fmp4 -f hls index.m3u8

Then combine the files into a single file by starting with the init:
cat init.mp4 > output.mp4
And then adding the rest of the files:
for f in index*.m4s; do echo $f; cat $f >> output.mp4; done

I would expect that the size and position of the keyframes would match up with the file sizes, but that's not the case:
ffprobe -show_entries packet=pos,flags -of csv=p=0 -i output.mp4 | grep K

Is there something I'm doing incorrectly or an assumption I'm making that's not correct?

[MasterQuestionable]

͏    "extra_keyframes_example.mp4":
͏    ​https://drive.usercontent.google.com/download?export=download&id=1WxOrSi-GNB45nLUUiR4PT7c4H2VurtKk
͏    (~ 18.34 MiB)

͏    Simple concatenation via `cat` alike seems to do garbage.
͏    Your commands also look problematic. (haven't detailedly verified)

[David Johansen]

Replying to MasterQuestionable:

͏    Simple concatenation via `cat` alike seems to do garbage.
͏    Your commands also look problematic. (haven't detailedly verified)

The resulting video plays correctly in QuickTime and VLC and isn't HLS playback of a fragmented MP4 doing this as well?

[MasterQuestionable]

͏    The structure of fMP4 shown in comment:11 linked clearly indicated something:
͏    That may be raw-concat'd is not the whole container, but only certain elements.

͏    FFmpeg currently may not handle non-perfectly-valid files in the most sensible way:
͏    ​https://trac.ffmpeg.org/ticket/10930#comment:5
͏    ​https://trac.ffmpeg.org/ticket/10008#comment:7
͏    ​https://trac.ffmpeg.org/ticket/11183#comment:1

͏    However, for your case: the application just seems to tolerate your error...

[David Johansen]

I'm definitely willing to adjust the way I process/create the files to do it the right way, so could you help me understand what needs to be adjusted to accomplish that?

From my understanding, Fragmented MPEG-4 is so the init section and the data are separated and can be combined like this for HLS playback, but is that incorrect?

[MasterQuestionable]

͏    I don't use HLS. So my knowledge on creating which would be limited.

͏    As for fMP4, highlight from the previous linked:
͏    ​https://i.sstatic.net/jc4x3.png

[David Johansen]

Yes, the generated files using the commands I sent above follow that and conform to the description of ​Fragmented MPEG-4 in the HLS spec with it using moof rather than moov