Regression with filter graph involving -shortest flag

[Wes Castro]

Summary of the bug:

A command line that worked with FFmpeg 5.1 produces a regressed output in FFmpeg 6.0, 6.1, and git-master. The expected output is a ~15 second MP4 with a video and audio stream. With git-master all video frames are dropped and only a 980ms audio stream is produced.

Apologies for the large filter graph... when attempting to reduce the chain I was unable to reproduce the issue in the same way.

How to reproduce:

Generate test inputs:

FFmpeg for all commands built with only ./configure --enable-zlib && make -j$(nproc)

ffmpeg -y -f lavfi -i testsrc=duration=15:size=1080x1920:rate=30 -f lavfi -i sine=frequency=1000:sample_rate=48000:duration=14.9 15secondvideoaudio.mp4
ffmpeg -y -f lavfi -i sine=frequency=1000:sample_rate=48000:duration=300 5minaudio.mp4
ffmpeg -y -f lavfi -i testsrc=size=1280x720 -frames:v 1 image.png

Run problematic filter graph:

ffmpeg -v 9 -loglevel 99 -hide_banner -y -i 15secondvideoaudio.mp4 -ss 21 -to 35 -i 5minaudio.mp4 -i 15secondvideoaudio.mp4 -framerate 30 -i image.png -framerate 30 -i image.png -filter_complex "[2:v] scale=w=1080:h=-2 [_fg_scaled] ; nullsrc=size=1080x1920:r=2:d=1, geq=red_expr='88+(Y/H)*(72-88)':green_expr='136+(Y/H)*(72-136)':blue_expr='184+(Y/H)*(72-184)', loop=-1:size=2, fps=fps=30 [_bg] ; [_bg][_fg_scaled] overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2:shortest=1 [uploaded_video_2_0_uploaded_video] ; [3:v][uploaded_video_2_0_uploaded_video] scale2ref=w=in_w*1.000000:h=in_h*1.000000:sws_flags=lanczos [_overlay][_video_in] ; [_video_in][_overlay] overlay=x=main_w*0.000000:y=main_h*0.000000:shortest=0:repeatlast=1 [video1] ; [4:v][video1] scale2ref=w=in_w*0.430556:h=in_h*0.075000:sws_flags=lanczos [_overlay][_video_in] ; [_video_in][_overlay] overlay=x=main_w*0.234429:y=main_h*-0.132120:shortest=0:repeatlast=1 [_video_out] ; [_video_out] setpts=PTS-STARTPTS [video]" -filter_complex "[1:a] volume=volume=1.000, afade=type=in:start_time=0:duration=0.4, afade=type=out:start_time=14.367:duration=0.6, aloop=loop=-1:size=2.14748e+09 [music] ; [0:a] volume=volume=0.000 [original_audio] ; [original_audio][music] amix=inputs=2:duration=longest, asetpts=PTS-STARTPTS [music_mix]" -map [music_mix] -map [video] -map_metadata -1 -shortest -f mp4 -y output.mp4

Results

• FFmpeg 5.1
  • Produces 15.04 second video+audio
• FFmpeg 6.0
  • Hangs indefinitely when encoding frame 446 (when I cancel it with Ctrl-C the last frame= line output is for 448)
• FFmpeg 6.1
  • Fails with Error while filtering: No space left
  • Workaround: Change loop=-1 to loop=99
    • Not ideal since loop is no longer infinite
    • Higher loop value (999) will cause conversion to fail with Error while filtering: No space left again
• FFmpeg master branch 3/26/2024 (9df11820650b1d93f6a49519aaa51afb5cc17086)
  • Warning is printed: [vost#0:1/mpeg4 @ 0x2e90500] No filtered frames for output stream, trying to initialize anyway.
  • Produces no video frames and 980ms of audio.

Notes

Changing the audio stream duration of 15secondvideoaudio.mp4 to 14.8 or 15.0 causes the hang to not occur in FFmpeg 6.0 or 6.1, but the master branch issue still occurs.

[Wes Castro]

Log files were uploaded with the VideoLAN file uploader.

[Michael Niedermayer]

Heres the history of commits changing the behavior of this testcase. Bisecting this was some work given how many times the behavior changed

master f5d2dc7b4bd2430c7a3d977b22bd20ef3508505c: only audio

ce178be8b0caf3255ceeeff14a8c3619dac5cc8a is the first bad commit
commit ce178be8b0caf3255ceeeff14a8c3619dac5cc8a
Author: Anton Khirnov <​anton@khirnov.net>
Date: Thu Mar 7 15:21:30 2024 +0100

fftools/ffmpeg_filter: do not assume av_buffersrc_get_nb_failed_requests()>0

Apparently it can happen that avfilter_graph_request_oldest() returns
EAGAIN, yet av_buffersrc_get_nb_failed_requests() returns 0 for every
input.

Works around #10795, though the root issue is most likely in the
scale2ref filter.

fftools/ffmpeg_filter.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Assertion best_input >= 0 failed at fftools/ffmpeg_filter.c:1922

commit 66e78e9680db2a0daecf4d79637dcbf872450f08
Author: Anton Khirnov <​anton@khirnov.net>
Date: Wed Jul 19 19:29:18 2023 +0200

fftools/ffmpeg_demux: switch from AVThreadMessageQueue to ThreadQueue

• the code is made shorter and simpler
• avoids constantly allocating and freeing AVPackets, thanks to ThreadQueue integration with ObjPool
• is consistent with decoding/filtering/muxing
• reduces the diff in the future switch to thread-aware scheduling

This makes ifile_get_packet() always block. Any potential issues caused
by this will be resolved by the switch to thread-aware scheduling in
future commits.

fftools/ffmpeg.c | 32 +++++++--------
fftools/ffmpeg.h | 3 +-
fftools/ffmpeg_demux.c | 106 +++++++++++++++++-------------------------------
fftools/ffmpeg_filter.c | 5 ++-
4 files changed, 57 insertions(+), 89 deletions(-)

Infinite loop

d9e41ead82263e96ebd14d4d88d6e7f858dd944c is the first bad commit
commit d9e41ead82263e96ebd14d4d88d6e7f858dd944c
Author: Paul B Mahol <​onemda@gmail.com>
Date: Fri Dec 1 16:59:07 2023 +0100

avfilter/avfilter: fix OOM case for default activate

Fixes OOM when caller keeps adding frames into filtergraph
that reached EOF by other means, for example EOF is signalled
by other filter in filtergraph or by buffersink.

fail no space left on device (loop 99 supposedly workarounds)

dde3de017088ab5bf7fb7920d3d8d4ba4e5dc290 is the first bad commit
commit dde3de017088ab5bf7fb7920d3d8d4ba4e5dc290
Author: Anton Khirnov <​anton@khirnov.net>
Date: Thu Jul 13 13:34:32 2023 +0200

fftools/ffmpeg_enc: return errors from do_*_out() instead of aborting

fftools/ffmpeg_enc.c | 29 ++++++++++++++---------------
1 file changed, 14 insertions(+), 15 deletions(-)

conversion failed

commit 923c6ab170669793d3311cbfd465a967d44f1636
Author: Anton Khirnov <​anton@khirnov.net>
Date: Fri Mar 24 08:59:05 2023 +0100

fftools/ffmpeg: use sync queues for enforcing audio frame size

The code currently uses lavfi for this, which creates a sort of
configuration dependency loop - the encoder should be ideally
initialized with information from the first audio frame, but to get this
frame one needs to first open the encoder to know the frame size. This
necessitates an awkward workaround, which causes audio handling to be
different from video.

With this change, audio encoder initialization is congruent with video.

fftools/ffmpeg.c | 58 ++++++++++-------------------------------------
fftools/ffmpeg_filter.c | 8 -------
fftools/ffmpeg_mux_init.c | 19 +++++++++++-----
3 files changed, 25 insertions(+), 60 deletions(-)

448 infinite

b2b9e9ccee0647b4695edaa66ae824850260ee02 is the first bad commit
commit b2b9e9ccee0647b4695edaa66ae824850260ee02
Author: Anton Khirnov <​anton@khirnov.net>
Date: Fri Jun 10 14:38:32 2022 +0200

fftools/ffmpeg: use last filter output pts to choose next output stream

This will be needed in following commits that will add new buffering
stages after encoding and bitstream filtering.

fftools/ffmpeg.c | 22 +++++++++++++++++-----
fftools/ffmpeg.h | 2 ++
fftools/ffmpeg_opt.c | 1 +
3 files changed, 20 insertions(+), 5 deletions(-)

5.1: working

[Michael Niedermayer]

Given that most commits changing the behavior are by Anton, i suggest to talk with Anton. Also the prior (working) avfilter timestamp/order/whatever you call it design is by Nicolas. So Nicolas opinion about this should be valuable too. I am not sure if Nicolas is available for consulting work (to fix this).

[Michael Niedermayer]

Adding anton to CC too

[Michael Niedermayer]

Also once this has been fixed, this testcase should be added as fate test, given how often behavior changed with apparently noone noticing.

[elenril]

This seems to be yet another symptom of the fundamental scale2ref sync problem which caused it to be deprecated in 95568c4e316e8f5f3252596b1f01ce1de22216b6. Replacing scale2ref in the above graph with scale (changing in_h/w to rh/w, and adding split=2 right before scale) produces a 15-second file for me, so closing this as fixed.