mpegts issue on localhost

[burek]

Hi,

In response to https://ffmpeg.org/trac/ffmpeg/ticket/602 (cehoyos), I'll try to provide my experience with mpegts format.

Here is the first issue. I have 2 ffmpeg processes, the first one should send the a/v (aac+h264) to the localhost:port using udp, and the 2nd ffmpeg process should pick it up and just save to the .ts file.

This is the command line of the first ffmpeg:

ffmpeg -y -re \
        -f video4linux2 -r 30 -s qvga -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 \
        -f alsa -ar 44100 -ac 2 -i default:CARD=Intel \
        -vcodec libx264 -crf 20 -threads 0 \
        -acodec aac -ab 32k -ar 44100 -ac 2 \
        -strict experimental \
        -vbsf h264_mp4toannexb \
        -absf aac_adtstoasc \
        -f mpegts udp://127.0.0.1:10001

And this is the command of the 2nd ffmpeg:

ffmpeg -muxdelay 15 -f mpegts -i udp://localhost:10001 -acodec copy -vcodec copy out.ts

Now, when I try to play the .ts file in VLC, I can watch the video, but in the message log I can see a lot of those:

main warning: late picture skipped (35805 > -268)
main warning: late picture skipped (2472 > -268)
main warning: late picture skipped (7218 > -262)
main warning: late picture skipped (1171 > -334)
main warning: late picture skipped (24818 > -266)
main warning: late picture skipped (19257 > -277)
main warning: late picture skipped (14977 > -303)
main warning: late picture skipped (57074 > -266)
main warning: late picture skipped (23740 > -266)
main warning: late picture skipped (5374 > -265)
main warning: late picture skipped (1929 > -15969)
main warning: late picture skipped (1376 > -264)

I'm sure the network is not an issue here :) and also, I've monitored cpu usage (on a dual core) and it didn't go over 40% at any time.

Also, when I record a live stream this way for a very long time (like 12 hours or more) the audio and video get totally async, meaning audio is progressively advancing before the video and after like 3-4 hours of recording, the drift is around 10 seconds and will get even bigger for the longer period of the recording.

I can send you samples of .ts files, only I'll need to record for that long, so you can see that it really happens.

Other issues are related to remuxing from .ts to .flv (also with h264+aac), but I'll see to create new tickets for that.

[Carl Eugen Hoyos]

Assuming this is not reproducible with

ffmpeg -re -f video4linux2 -r 30 -s qvga -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 -vcodec libx264 -crf 20 -vbsf h264_mp4toannexb out.ts

Isn't this more an udp issue?

And why are you using the bsf?

[burek]

Well, I didn't try just recording to .ts file for that long, more rational way was to stream to an udp port because that way I could duplicate the data and watch it, while it is being saved to a file.

Anyway, for the absf, I don't remember now where exactly did I pick up that option (was it in ffmpeg channel or google), I just know that without it, when I play the udp stream in VLC, then the audio is a lot more quirky, skipping up/down-sampling and stuff... And since there is no official documentation on that filter ( ​http://ffmpeg.org/ffmpeg.html#aac_005fadtstoasc ), I didn't have chance to read about its real intended usage, so it might be the reason of the async behavior.

Can You give me one working example (default or something) of using mpegts (h264+aac) over udp, just to test it, because I've tried so many different combinations and variations I can't even remember now and none of them gave the smooth and clean playback without async (I'm talking about 100/100 LAN network). What could be the simplest way to start such a stream?

[Carl Eugen Hoyos]

I meant the video bitstream filter that should only make sense when stream copying from mp4.

To clarify:
Does creating the transport stream without using udp (and without audio and without the video bitstream filter) produce a file that does not show the warnings in vlc?

[burek]

why without audio?

[burek]

encoding audio only works just fine:

ffmpeg -y -re \
        -f alsa -ar 44100 -ac 2 -i default:CARD=Intel \
        -acodec aac -ab 32k -ar 44100 -ac 2 \
        -strict experimental \
        out.ts

No warnings, errors, nothin, just clear audio :)

[Carl Eugen Hoyos]

Is this only reproducible with an external library (-vcodec libx264) or also with an internal encoder?

Please post complete, uncut console output (including command line) without -threads 0.

And do you have any idea what the vlc warnings mean? Are there any visible problems with the resulting file? Do other players (ffplay, MPlayer, WMP 12) show any problems?

[burek]

Same scenario with this example (mpeg2video):

ffmpeg -y -re \
        -f video4linux2 -r 30 -s qvga -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 \
        out.ts

ffmpeg version N-34519-g54a09f1, Copyright (c) 2000-2011 the FFmpeg developers
  built on Nov  6 2011 12:46:05 with gcc 4.6.1
  configuration: --enable-static --enable-shared --enable-gpl --enable-nonfree --enable-postproc --enable-libx264 --enable-libaacplus --enable-libmp3lame --enable-libv4l2 --enable-libvo-aacenc --enable-version3
  libavutil    51. 23. 0 / 51. 23. 0
  libavcodec   53. 28. 0 / 53. 28. 0
  libavformat  53. 19. 0 / 53. 19. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 47. 0 /  2. 47. 0
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[video4linux2,v4l2 @ 0x1f9c980] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0':
  Duration: N/A, start: 27501.143731, bitrate: 27648 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27648 kb/s, 30 tbr, 1000k tbn, 30 tbc
[buffer @ 0x1f9c800] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[mpegts @ 0x1f97040] muxrate VBR, pcr every 3 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'out.ts':
  Metadata:
    encoder         : Lavf53.19.0
    Stream #0:0: Video: mpeg2video, yuv420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0 (rawvideo -> mpeg2video)
Press [q] to stop, [?] for help
frame=  976 fps= 30 q=5.7 Lsize=    1257kB time=00:00:32.50 bitrate= 317.0kbits/s dup=638 drop=0
video:1070kB audio:0kB global headers:0kB muxing overhead 17.558359%

And here is libx264 without -threads:

# ffmpeg -y -re \
        -f video4linux2 -r 30 -s qvga -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 \
        -vcodec libx264 -crf 20 \
        out.ts

ffmpeg version N-34519-g54a09f1, Copyright (c) 2000-2011 the FFmpeg developers
  built on Nov  6 2011 12:46:05 with gcc 4.6.1
  configuration: --enable-static --enable-shared --enable-gpl --enable-nonfree --enable-postproc --enable-libx264 --enable-libaacplus --enable-libmp3lame --enable-libv4l2 --enable-libvo-aacenc --enable-version3
  libavutil    51. 23. 0 / 51. 23. 0
  libavcodec   53. 28. 0 / 53. 28. 0
  libavformat  53. 19. 0 / 53. 19. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 47. 0 /  2. 47. 0
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[video4linux2,v4l2 @ 0x707980] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0':
  Duration: N/A, start: 27685.969253, bitrate: 27648 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27648 kb/s, 30 tbr, 1000k tbn, 30 tbc
[buffer @ 0x708440] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 @ 0x7025a0] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
[libx264 @ 0x7025a0] profile High, level 1.3
[mpegts @ 0x702040] muxrate VBR, pcr every 3 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'out.ts':
  Metadata:
    encoder         : Lavf53.19.0
    Stream #0:0: Video: h264, yuv420p, 320x240, q=-1--1, 90k tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0 (rawvideo -> libx264)
Press [q] to stop, [?] for help
frame=  636 fps= 29 q=-1.0 Lsize=     626kB time=00:00:21.13 bitrate= 242.5kbits/s dup=442 drop=0
video:497kB audio:0kB global headers:0kB muxing overhead 25.792506%
[libx264 @ 0x7025a0] frame I:3     Avg QP:14.98  size: 15692
[libx264 @ 0x7025a0] frame P:424   Avg QP:19.03  size:  1074
[libx264 @ 0x7025a0] frame B:209   Avg QP:25.71  size:    33
[libx264 @ 0x7025a0] consecutive B-frames: 56.1%  0.0%  0.5% 43.4%
[libx264 @ 0x7025a0] mb I  I16..4:  2.4% 62.2% 35.3%
[libx264 @ 0x7025a0] mb P  I16..4:  0.0%  0.1%  0.0%  P16..4: 33.9%  9.4%  7.7%  0.0%  0.0%    skip:48.9%
[libx264 @ 0x7025a0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  3.4%  0.0%  0.0%  direct: 0.2%  skip:96.3%  L0:54.6% L1:44.0% BI: 1.4%
[libx264 @ 0x7025a0] 8x8 transform intra:64.4% inter:74.7%
[libx264 @ 0x7025a0] coded y,uvDC,uvAC intra: 97.3% 99.6% 98.7% inter: 9.6% 30.5% 18.6%
[libx264 @ 0x7025a0] i16 v,h,dc,p:  3%  3% 29% 65%
[libx264 @ 0x7025a0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 20% 13%  3%  9%  5% 17%  5%  9%
[libx264 @ 0x7025a0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 20%  8%  5% 10%  6% 15%  9%  9%
[libx264 @ 0x7025a0] i8c dc,h,v,p: 62% 19% 10%  9%
[libx264 @ 0x7025a0] Weighted P-Frames: Y:0.7% UV:0.2%
[libx264 @ 0x7025a0] ref P L0: 63.4%  6.8%  7.9% 21.7%  0.1%
[libx264 @ 0x7025a0] ref B L0: 85.1% 13.1%  1.8%
[libx264 @ 0x7025a0] ref B L1: 91.4%  8.6%
[libx264 @ 0x7025a0] kb/s:192.20

The problem still exists.
That message tells that the timestamps are somehow wrong, I guess. Sometimes I can see the image freezes for a fraction of a second and the image goes distorted just like a wrong keyframe was used for all the subsequent delta frames, but it gets back to normal after a couple of seconds.

[Carl Eugen Hoyos]

Are the problems only reproducible with -vcodec mpeg2video -f mpegts or also for -vcodec mpeg2video -f avi/mpg/mkv ?

[burek]

-vcodec mpeg2video -f avi: no warnings at all
-vcodec mpeg2video -f mpeg: warnings are there constantly, just like with mpegts
-vcodec mpeg2video -f matroska: no warnings at all

[Carl Eugen Hoyos]

Is the problem also reproducible with

ffmpeg -f video4linux2 -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 out.ts

?
Or is -r 30 needed to make the conversion work at all?

[burek]

-r 30 just tells the driver to correctly sample the video input, so we don't have to resample it afterwards

here is the log:

# ffmpeg -f video4linux2 -i /dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0 out.ts
ffmpeg version N-34519-g54a09f1, Copyright (c) 2000-2011 the FFmpeg developers
  built on Nov  6 2011 12:46:05 with gcc 4.6.1
  configuration: --enable-static --enable-shared --enable-gpl --enable-nonfree --enable-postproc --enable-libx264 --enab                                     le-libaacplus --enable-libmp3lame --enable-libv4l2 --enable-libvo-aacenc --enable-version3
  libavutil    51. 23. 0 / 51. 23. 0
  libavcodec   53. 28. 0 / 53. 28. 0
  libavformat  53. 19. 0 / 53. 19. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 47. 0 /  2. 47. 0
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[video4linux2,v4l2 @ 0x1daf780] The driver changed the time per frame from 32767/1996117544 to 1/30
[video4linux2,v4l2 @ 0x1daf780] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/v4l/by-id/usb-046d_0809_33A881A3-video-index0':
  Duration: N/A, start: 25528.008073, bitrate: 27648 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27648 kb/s, 30 tbr, 1000k tbn, 30 tbc
[buffer @ 0x1dbdf40] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[mpegts @ 0x1dbd0a0] muxrate VBR, pcr every 3 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'out.ts':
  Metadata:
    encoder         : Lavf53.19.0
    Stream #0:0: Video: mpeg2video, yuv420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0 (rawvideo -> mpeg2video)
Press [q] to stop, [?] for help
frame=  480 fps= 37 q=10.3 Lsize=     754kB time=00:00:15.96 bitrate= 386.9kbits/s dup=287 drop=0
video:658kB audio:0kB global headers:0kB muxing overhead 14.622744%

and warnings are also there again :/

[burek]

I apologize, it seems that this error is related to a slow hardware (??) although I'm not sure why, because the cpu usage never gets over 30-40%, but they say its related to a slow decoder/cpu or something like that, so this bug report can be closed without further examination, I guess.