Dropped frames if x11grab and webcam are used in filter_complex

[Ian Milligan]

Using the overlay filter to overlay video from a webcam onto x11grab causes many dropped frames. The frames seem to come in short bursts followed by a large number of dropped frames. This is easily seen by inserting showinfo into the x11grab stream before it is sent to the overlay filter. The attached file shows this behaviour. If the overlay filter isn't used, the frames appear to have approximately the correct timestamps as shown in the second attachment.

[Carl Eugen Hoyos]

What happens if you significantly decrease the x11grab resolution (640x480 or 400x200)?

[Ian Milligan]

I get the same behavior, even at 400x200.

[Clément Bœsch]

Wouldn't that be related to the nanosleep hack used to sync with the framerate?

[Ian Milligan]

I wrote the attached patch to make x11grab respect AVFMT_FLAGS_NONBLOCK but it didn't seem to fix the issue.

[Carl Eugen Hoyos]

Is the problem also reproducible if you replace -f x11grab -video_size 1680x1050 -framerate 30 -i :0.0 in your command line with -f lavfi -i testsrc=s=1680x1050:r=30?

The following works fine here:

$ ffmpeg -f x11grab -s 1920x1200 -i :0 -f v4l2 -s 320x176 -i /dev/video1 -filter_complex overlay -qscale 2 out.avi

[Ian Milligan]

No, if I use the testsrc instead of x11grab it seems to work.

[Carl Eugen Hoyos]

frame= 551 fps= 22 q=-1.0 Lsize= 3615kB time=00:00:24.63 bitrate=1202.2kbits/s

As pointed out on the mailing list, this indicates the problem is also reproducible without overlay: Your system does not reach the requested frame rate. Try with overlay and a framerate of ~15.

[Ian Milligan]

Lowering the framerate does not seem to help. It seems that in the nonoverlay version, it takes a second or so for everything to initialize so the framerate x264 sees starts low and converges towards 30. In the overlay version the framerate is stuck at around 5-6 fps.

[jnvsor]

Switching the input order is a workaround

ffmpeg \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-f v4l2 -framerate 30 -i /dev/video0 \
-filter_complex '[0][1]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  225 fps= 18 q=-1.0 Lsize=   36782kB time=00:00:12.56 bitrate=23977.3kbits/s dup=0 drop=74

Switch the input order and switch it on the overlay filter again:

ffmpeg \
-f v4l2 -framerate 30 -i /dev/video0 \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-filter_complex '[1][0]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  208 fps= 30 q=-1.0 Lsize=   25993kB time=00:00:06.93 bitrate=30711.3kbits/s

[jnvsor]

I can confirm it occurs when both streams are passed through filter_complex, even if they are null filters that shouldn't effect performance at all.

Low fps:

ffmpeg -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0 -filter_complex '[0]null[cap];[1]null[webcam]' -map '[cap]' -map '[webcam]' -c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 -threads 2 -y out.avi

Remove either of the null filters and it will work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[1]null[webcam]'
+ -map 0
OR
+ '[0]null[cap]'
+ -map 1

Add the v4l2 input first and it will work at full framerate:

- -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0
+ -f v4l2 -framerate 30 -i /dev/video0 -f x11grab -framerate 30 -s hd1080 -i :0.0

It only occurs if that filter output is actually used, piping one to a nullsink also makes it work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[0]null[cap];[cap]nullsink;[1]null[webcam]'
+ -map 0

[jnvsor]

Changing the title since it can be reproduced without overlay, it just so happens overlay requires both streams to be passed through filter_complex which seems to be closer to the root of the problem

ffmpeg \
-f x11grab -i :0.0 \
-f v4l2 -i /dev/video0 \
-filter_complex '[0]null[cap];[1]null[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -y out.mkv

[jnvsor]

Happens with webcam audio too!

ffmpeg \
-f x11grab -i :0.0 \
-f pulse -i alsa_input.usb-046d_0802_F1851F50-02-U0x46d0x802.analog-mono -channel_layout stereo \
-filter_complex '[0]null[cap];[1]anull[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -c:a libmp3lame -y test.mkv