filter_complex losing frames

[Takis Issaris]

Summary of the bug:
I'm overlaying transparant PNGs over another video which I'm upscaling. Somehow I seem to be losing frames.

How to reproduce:

% ffmpeg  -i big_buck_bunny_1080p_h264.mov -i frame%05d.png  -filter_complex "[0:v] scale=3840:2160[bbb];[bbb][1:v] overlay[out]" -map "[out]" test.mp4
ffmpeg version N-44322-g1e9caf9
built on Sep 10 2012 11:35:03 with clang version 3.1 (tags/RELEASE_31/final)

[Takis Issaris]

Full output:

ffmpeg  -i /srv/files/videos/big_buck_bunny_1080p_h264.mov -i frame%05d.png  -filter_complex "[0:v] scale=3840:2160[bbb];[bbb][1:v] overlay[out]" -map "[out]" /usr/local/src/test.mp4
ffmpeg version N-44322-g1e9caf9 Copyright (c) 2000-2012 the FFmpeg developers
  built on Sep 10 2012 11:35:03 with clang version 3.1 (tags/RELEASE_31/final)
  configuration: --cc=/opt/clang/bin/clang --enable-avfilter --enable-frei0r --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-nonfree --enable-pthreads --disable-shared --enable-static --enable-version3 --enable-x11grab
  libavutil      51. 72.100 / 51. 72.100
  libavcodec     54. 55.100 / 54. 55.100
  libavformat    54. 25.105 / 54. 25.105
  libavdevice    54.  2.100 / 54.  2.100
  libavfilter     3. 16.101 /  3. 16.101
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/srv/files/videos/big_buck_bunny_1080p_h264.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2008-05-27 18:40:35
    timecode        : 00:00:00:00
  Duration: 00:09:56.46, start: 0.000000, bitrate: 4849 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080, 9282 kb/s, 24 fps, 24 tbr, 2400 tbn, 4800 tbc
    Metadata:
      creation_time   : 2008-05-27 18:40:35
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      creation_time   : 2008-05-27 18:40:35
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
    Stream #0:2(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, 5.1, s16, 437 kb/s
    Metadata:
      creation_time   : 2008-05-27 18:40:35
      handler_name    : Apple Alias Data Handler
Input #1, image2, from 'frame%05d.png':
  Duration: 00:00:20.00, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: png, rgba, 3840x2160, 25 fps, 25 tbr, 25 tbn, 25 tbc
File '/usr/local/src/test.mp4' already exists. Overwrite ? [y/N] y
264 - core 125 r2200 999b753 - H.264/MPEG-4 AVC codec - Copyleft 2003-2012 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=24 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/usr/local/src/test.mp4':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    timecode        : 00:00:00:00
    encoder         : Lavf54.25.105
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 3840x2160, q=-1--1, 24 tbn, 24 tbc
Stream mapping:
  Stream #0:0 (h264) -> scale
  Stream #1:0 (png) -> overlay:overlay
  overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
frame=   34 fps=2.2 q=-1.0 Lsize=    1118kB time=00:00:01.33 bitrate=6866.6kbits/s    
video:1117kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.086493%

[Takis Issaris]

I noticed that I was losing frames as the numbered%05d.png's contain the framenumber in each frame.

[Cigaes]

Replying to takis:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/srv/files/videos/big_buck_bunny_1080p_h264.mov':
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080, 9282 kb/s, 24 fps, 24 tbr, 2400 tbn, 4800 tbc
Input #1, image2, from 'frame%05d.png':
    Stream #1:0: Video: png, rgba, 3840x2160, 25 fps, 25 tbr, 25 tbn, 25 tbc

As quoted, your main video is 24 FPS while the overlay video (they are actually images, but ffmpeg sees them as a single video) is 25 FPS: one frame per second of the overlay falls between two frames of the main input and is dropped. That is normal behaviour. If your frames are supposed to match, specify the frame rate for the images.

[Takis Issaris]

Ah! I should have read FFmpeg's output... I did not know it assumed the image sequence would be a 25 fps video.

Thanks for your fast reply!

[Takis Issaris]

Here's the updated commandline:

ffmpeg  -i big_buck_bunny_1080p_h264.mov -f image2 -r 24 -i frame%05d.png  -filter_complex "[0:v] scale=3840:2160[bbb];[bbb][1:v] overlay[out]" -map "[out]" test.mp4

Leaving out the "-f image2" does not work though. Not sure why.

Unfortunately, there's still one frame missing: The second frame always goes missing now.

[Cigaes]

This particular file is missing its "second" frame, i.e., the frame at pts=1/24, just between pts=0/24 and pts=2/24:

$ ffprobe -loglevel quiet -show_frames -of compact /tmp/big_buck_bunny_1080p_h264.mov
frame|media_type=video|key_frame=1|pkt_pts=0|pkt_pts_time=0.000000|pkt_dts=-100|pkt_dts_time=-0.041667|pkt_duration=100|pkt_duration_time=0.041667|pkt_pos=441422|width=1920|height=1080|pix_fmt=yuv420p|sample_aspect_ratio=N/A|pict_type=I|coded_picture_number=0|display_picture_number=0|interlaced_frame=0|top_field_first=0|repeat_pict=0|reference=0
frame|media_type=video|key_frame=0|pkt_pts=200|pkt_pts_time=0.083333|pkt_dts=0|pkt_dts_time=0.000000|pkt_duration=100|pkt_duration_time=0.041667|pkt_pos=447743|width=1920|height=1080|pix_fmt=yuv420p|sample_aspect_ratio=N/A|pict_type=P|coded_picture_number=1|display_picture_number=0|interlaced_frame=0|top_field_first=0|repeat_pict=0|reference=0
frame|media_type=video|key_frame=0|pkt_pts=300|pkt_pts_time=0.125000|pkt_dts=300|pkt_dts_time=0.125000|pkt_duration=100|pkt_duration_time=0.041667|pkt_pos=592358|width=1920|height=1080|pix_fmt=yuv420p|sample_aspect_ratio=N/A|pict_type=B|coded_picture_number=4|display_picture_number=0|interlaced_frame=0|top_field_first=0|repeat_pict=0|reference=0
frame|media_type=video|key_frame=0|pkt_pts=400|pkt_pts_time=0.166667|pkt_dts=400|pkt_dts_time=0.166667|pkt_duration=100|pkt_duration_time=0.041667|pkt_pos=448026|width=1920|height=1080|pix_fmt=yuv420p|sample_aspect_ratio=N/A|pict_type=P|coded_picture_number=3|display_picture_number=0|interlaced_frame=0|top_field_first=0|repeat_pict=0|reference=3

You can force it to be duplicated, and therefore available for overlay by inserting the fps=fps=24 near the input (just after the scale, probably).

[Carl Eugen Hoyos]

Is this a valid ticket or can it be closed?