Opened 11 years ago

Last modified 11 years ago

#3192 new defect

Dropped frames if x11grab and webcam are used in filter_complex

Reported by: Ian Milligan Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords: x11grab
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Using the overlay filter to overlay video from a webcam onto x11grab causes many dropped frames. The frames seem to come in short bursts followed by a large number of dropped frames. This is easily seen by inserting showinfo into the x11grab stream before it is sent to the overlay filter. The attached file shows this behaviour. If the overlay filter isn't used, the frames appear to have approximately the correct timestamps as shown in the second attachment.

Attachments (3)

ffmpeg-overlay.log (61.5 KB ) - added by Ian Milligan 11 years ago.
ffmpeg-nooverlay.log (214.5 KB ) - added by Ian Milligan 11 years ago.
x11grab-nonblock.diff (1.9 KB ) - added by Ian Milligan 11 years ago.

Download all attachments as: .zip

Change History (17)

by Ian Milligan, 11 years ago

Attachment: ffmpeg-overlay.log added

by Ian Milligan, 11 years ago

Attachment: ffmpeg-nooverlay.log added

comment:1 by Carl Eugen Hoyos, 11 years ago

What happens if you significantly decrease the x11grab resolution (640x480 or 400x200)?

comment:2 by Ian Milligan, 11 years ago

I get the same behavior, even at 400x200.

comment:3 by Clément Bœsch, 11 years ago

Wouldn't that be related to the nanosleep hack used to sync with the framerate?

by Ian Milligan, 11 years ago

Attachment: x11grab-nonblock.diff added

comment:4 by Ian Milligan, 11 years ago

I wrote the attached patch to make x11grab respect AVFMT_FLAGS_NONBLOCK but it didn't seem to fix the issue.

comment:5 by Carl Eugen Hoyos, 11 years ago

Is the problem also reproducible if you replace -f x11grab -video_size 1680x1050 -framerate 30 -i :0.0 in your command line with -f lavfi -i testsrc=s=1680x1050:r=30?

The following works fine here:

$ ffmpeg -f x11grab -s 1920x1200 -i :0 -f v4l2 -s 320x176 -i /dev/video1 -filter_complex overlay -qscale 2 out.avi

comment:6 by Ian Milligan, 11 years ago

No, if I use the testsrc instead of x11grab it seems to work.

comment:7 by Carl Eugen Hoyos, 11 years ago

frame= 551 fps= 22 q=-1.0 Lsize= 3615kB time=00:00:24.63 bitrate=1202.2kbits/s

As pointed out on the mailing list, this indicates the problem is also reproducible without overlay: Your system does not reach the requested frame rate. Try with overlay and a framerate of ~15.

comment:8 by Ian Milligan, 11 years ago

Lowering the framerate does not seem to help. It seems that in the nonoverlay version, it takes a second or so for everything to initialize so the framerate x264 sees starts low and converges towards 30. In the overlay version the framerate is stuck at around 5-6 fps.

comment:9 by Carl Eugen Hoyos, 11 years ago

Keywords: x11grab added

comment:10 by jnvsor, 11 years ago

Switching the input order is a workaround

ffmpeg \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-f v4l2 -framerate 30 -i /dev/video0 \
-filter_complex '[0][1]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  225 fps= 18 q=-1.0 Lsize=   36782kB time=00:00:12.56 bitrate=23977.3kbits/s dup=0 drop=74

Switch the input order and switch it on the overlay filter again:

ffmpeg \
-f v4l2 -framerate 30 -i /dev/video0 \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-filter_complex '[1][0]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  208 fps= 30 q=-1.0 Lsize=   25993kB time=00:00:06.93 bitrate=30711.3kbits/s

comment:11 by jnvsor, 11 years ago

I can confirm it occurs when both streams are passed through filter_complex, even if they are null filters that shouldn't effect performance at all.

Low fps:

ffmpeg -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0 -filter_complex '[0]null[cap];[1]null[webcam]' -map '[cap]' -map '[webcam]' -c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 -threads 2 -y out.avi

Remove either of the null filters and it will work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[1]null[webcam]'
+ -map 0
OR
+ '[0]null[cap]'
+ -map 1

Add the v4l2 input first and it will work at full framerate:

- -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0
+ -f v4l2 -framerate 30 -i /dev/video0 -f x11grab -framerate 30 -s hd1080 -i :0.0

It only occurs if that filter output is actually used, piping one to a nullsink also makes it work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[0]null[cap];[cap]nullsink;[1]null[webcam]'
+ -map 0

comment:12 by Carl Eugen Hoyos, 11 years ago

Summary: Dropped Frames With Overlay Filterx11grab drops frames if overlay filter is used

comment:13 by jnvsor, 11 years ago

Summary: x11grab drops frames if overlay filter is usedDropped frames if v4l2 and x11grab are used in filter_complex

Changing the title since it can be reproduced without overlay, it just so happens overlay requires both streams to be passed through filter_complex which seems to be closer to the root of the problem

ffmpeg \
-f x11grab -i :0.0 \
-f v4l2 -i /dev/video0 \
-filter_complex '[0]null[cap];[1]null[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -y out.mkv

comment:14 by jnvsor, 11 years ago

Summary: Dropped frames if v4l2 and x11grab are used in filter_complexDropped frames if x11grab and webcam are used in filter_complex

Happens with webcam audio too!

ffmpeg \
-f x11grab -i :0.0 \
-f pulse -i alsa_input.usb-046d_0802_F1851F50-02-U0x46d0x802.analog-mono -channel_layout stereo \
-filter_complex '[0]null[cap];[1]anull[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -c:a libmp3lame -y test.mkv
Note: See TracTickets for help on using tickets.