Fix Non-monotonous DTS for flv live stream output

[FishB8]

I'm running into problems when using ffmpeg to pull a live rtmp stream, re-encode it, and push the result back out to another location. The problem is that the dts can jump when the source stream is switched within the server that is providing the source for ffmpeg. I would assume that +genpts and +igndts flags should fix the problem, but they do not.

When the streams get switched internally in the source rtmp server, I get a bunch of:

[flv @ 0xdeadbeef] Non-monotonous DTS in output stream [blah blah blah ...]

% ffmpeg -i rtmp://0.0.0.0/foo/bar -fflags +genpts+igndts [...] -f flv -rtmp_live live rtmp://0.0.0.0/hash/bang

ffmpeg needs a means to deal with large jumps in dts gracefully. Currently it totally screws up the output stream. Ideally the origional dts would be tossed and new one generated when re-encoding based off the frame rate / frame base. Even better would be a bitstream filter to rewrite dts that could be used even in cases where you are doing a stream copy instead of re-encoding.

ffmpeg version 2.2.1 (using internal RTMP implementation)

[Carl Eugen Hoyos]

Is the problem also reproducible with file input instead of network input?
Is the problem also reproducible with file output instead of network output?
Please test current git head and please provide your failing command line together with the complete, uncut console output to make this a valid ticket.

[FishB8]

Note: to avoid confusion, from here on I'll use "network input/output" in reference to rtmp IO, while "stream" refers to video/audio/data element streams within the format.

I built latest git, no difference

The problem is reproducible with both file and network output. Basically the audio continues, but the video is stopped waiting for a frame with the correct DTS value.

Same thing happens when the source is a file with DTS discontinuity rather than a network input.

Once caveat is that if using stream copy for the codecs rather than re-encoding, the video output seems to continue once the DTS discontinuity occurs, but at a fraction of the proper frame rate. (Audio continues normally)

Here's the commandline / console output:

$ ffmpeg -fflags +genpts -i rtmp://127.0.0.1/switcher/output -c:a libfdk_aac -vbr 3 -ac 2 -ar 48000 -c:v libx264 -g 31 -crf 22 -profile:v high -preset:v medium -trellis 2 -f flv -rtmp_live live rtmp://127.0.0.1/publish/program
ffmpeg version N-62374-ge89f3d0 Copyright (c) 2000-2014 the FFmpeg developers
  built on Apr 12 2014 11:08:18 with gcc 4.8.2 (Gentoo 4.8.2 p1.3r1, pie-0.5.8r1)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange' --extra-cflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange' --extra-cxxflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange -fvisibility-inlines-hidden -DNDEBUG' --disable-static --enable-gpl --enable-postproc --enable-avfilter --enable-avresample --disable-stripping --enable-nonfree --enable-version3 --enable-nonfree --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=oss --disable-outdev=oss --enable-version3 --enable-nonfree --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --enable-gnutls --enable-hardcoded-tables --enable-iconv --enable-network --enable-openssl --enable-ffplay --enable-vaapi --enable-vdpau --enable-xlib --enable-zlib --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libmp3lame --enable-libaacplus --enable-libfaac --enable-libtheora --enable-libtwolame --disable-libwavpack --enable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --enable-libcdio --enable-libiec61883 --enable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --disable-libv4l2 --enable-libpulse --enable-x11grab --disable-libflite --enable-frei0r --enable-fontconfig --enable-ladspa --disable-libass --enable-libfreetype --disable-libsoxr --enable-pthreads --enable-libopencore-amrwb --enable-libopencore-amrnb --enable-libfdk-aac --enable-libopenjpeg --enable-libbluray --disable-libcelt --disable-libgme --enable-libgsm --enable-libmodplug --enable-libopus --disable-libquvi --disable-librtmp --disable-libssh --enable-libschroedinger --enable-libspeex --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-mips32r2 --disable-mipsdspr1 --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-sse42 --cpu=host
  libavutil      52. 75.100 / 52. 75.100
  libavcodec     55. 58.103 / 55. 58.103
  libavformat    55. 37.100 / 55. 37.100
  libavdevice    55. 13.100 / 55. 13.100
  libavfilter     4.  4.100 /  4.  4.100
  libavresample   1.  2.  0 /  1.  2.  0
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
[flv @ 0xd06450] Stream discovered after head already parsed
Input #0, flv, from 'rtmp://127.0.0.1/switcher/output':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 29
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 837.824000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 30.33 fps, 29.97 tbr, 1k tbn, 59.94 tbc
    Stream #0:1: Audio: aac, 48000 Hz, stereo, fltp
    Stream #0:2: Data: none
[libx264 @ 0xe99550] using SAR=1/1
[libx264 @ 0xe99550] using cpu capabilities: MMX2 SSE2Fast LZCNT
[libx264 @ 0xe99550] profile High, level 3.1
[libx264 @ 0xe99550] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=31 keyint_min=3 scenecut=40 intra_refresh=0 rc_lookahead=31 rc=crf mbtree=1 crf=22.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libfdk_aac @ 0xe95130] Note, the VBR setting is unsupported and only works with some parameter combinations
Output #0, flv, to 'rtmp://127.0.0.1/publish/program':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 29
    profile         : 
    level           : 
    encoder         : Lavf55.37.100
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 1k tbn, 29.97 tbc
    Stream #0:1: Audio: aac (libfdk_aac) ([10][0][0][0] / 0x000A), 48000 Hz, stereo, s16
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libfdk_aac)
Press [q] to stop, [?] for help
[h264 @ 0xd0f570] AVC: nal size 23330847kB time=00:00:03.92 bitrate= 484.4kbits/s    
[h264 @ 0xd0f570] missing picture in access unit with size 35                                                                                                                                                                            
[flv @ 0xd06450] DTS 0 < 841784 out of order
[libfdk_aac @ 0xe95130] Queue input is backward in time                                                                                                                                                                                  
[flv @ 0x141dd70] Non-monotonous DTS in output stream 0:1; previous: 3947, current: -837781; changing to 3947. This may result in incorrect timestamps in the output file.

(Repeats flv DTS warning for each frame) 

I can provide example files (both source and output) and an nginx-rtmp config file to re-create this issue if needed.

[Carl Eugen Hoyos]

The problem is reproducible with both file and network output.

Same thing happens when the source is a file with DTS discontinuity rather than a network input.

Please understand that issues involving the network are a magnitude more difficult to reproduce than bugs concerning files so please provide an input file sample (and a failing command line for that sample including complete, uncut console output) to allow us to reproduce the problem.

[FishB8]

Ok, I'm going attach an example file. The following is the result of trying to do the exact same as was shown previously, just using files instead of network rtmp IO.

ffmpeg -fflags +genpts+igndts -i DTS_Discontinuity.flv -c:a libfdk_aac -vbr 3 -ac 2 -ar 48000 -c:v libx264 -g 31 -crf 22 -profile:v high -preset:v medium -trellis 2 -f flv DTS_Test.flv 
ffmpeg version N-62374-ge89f3d0 Copyright (c) 2000-2014 the FFmpeg developers
  built on Apr 12 2014 11:08:18 with gcc 4.8.2 (Gentoo 4.8.2 p1.3r1, pie-0.5.8r1)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange' --extra-cflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange' --extra-cxxflags='-O2 -march=native -mfpmath=sse -ftree-vectorize -fomit-frame-pointer -pipe -fprefetch-loop-arrays -floop-block -floop-strip-mine -floop-interchange -fvisibility-inlines-hidden -DNDEBUG' --disable-static --enable-gpl --enable-postproc --enable-avfilter --enable-avresample --disable-stripping --enable-nonfree --enable-version3 --enable-nonfree --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=oss --disable-outdev=oss --enable-version3 --enable-nonfree --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --enable-gnutls --enable-hardcoded-tables --enable-iconv --enable-network --enable-openssl --enable-ffplay --enable-vaapi --enable-vdpau --enable-xlib --enable-zlib --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libmp3lame --enable-libaacplus --enable-libfaac --enable-libtheora --enable-libtwolame --disable-libwavpack --enable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --enable-libcdio --enable-libiec61883 --enable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --disable-libv4l2 --enable-libpulse --enable-x11grab --disable-libflite --enable-frei0r --enable-fontconfig --enable-ladspa --disable-libass --enable-libfreetype --disable-libsoxr --enable-pthreads --enable-libopencore-amrwb --enable-libopencore-amrnb --enable-libfdk-aac --enable-libopenjpeg --enable-libbluray --disable-libcelt --disable-libgme --enable-libgsm --enable-libmodplug --enable-libopus --disable-libquvi --disable-librtmp --disable-libssh --enable-libschroedinger --enable-libspeex --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-mips32r2 --disable-mipsdspr1 --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-sse42 --cpu=host
  libavutil      52. 75.100 / 52. 75.100
  libavcodec     55. 58.103 / 55. 58.103
  libavformat    55. 37.100 / 55. 37.100
  libavdevice    55. 13.100 / 55. 13.100
  libavfilter     4.  4.100 /  4.  4.100
  libavresample   1.  2.  0 /  1.  2.  0
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, flv, from 'DTS_Discontinuity.flv':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 29
    profile         : 
    level           : 
  Duration: 00:00:06.28, start: 8.854000, bitrate: 4266 kb/s
    Stream #0:0: Video: h264 (High), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 30.33 fps, 29.97 tbr, 1k tbn, 59.94 tbc
    Stream #0:1: Audio: aac, 48000 Hz, stereo, fltp
File 'DTS_Test.flv' already exists. Overwrite ? [y/N] y
[libx264 @ 0x1c5b1f0] using SAR=1/1
[libx264 @ 0x1c5b1f0] using cpu capabilities: MMX2 SSE2Fast LZCNT
[libx264 @ 0x1c5b1f0] profile High, level 3.1
[libx264 @ 0x1c5b1f0] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=31 keyint_min=3 scenecut=40 intra_refresh=0 rc_lookahead=31 rc=crf mbtree=1 crf=22.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libfdk_aac @ 0x1c5c200] Note, the VBR setting is unsupported and only works with some parameter combinations
Output #0, flv, to 'DTS_Test.flv':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 29
    profile         : 
    level           : 
    encoder         : Lavf55.37.100
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 1k tbn, 29.97 tbc
    Stream #0:1: Audio: aac (libfdk_aac) ([10][0][0][0] / 0x000A), 48000 Hz, stereo, s16
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libfdk_aac)
Press [q] to stop, [?] for help
[h264 @ 0x15697a0] AVC: nal size 23330847B time=00:00:04.69 bitrate= 247.0kbits/s    
[h264 @ 0x15697a0] missing picture in access unit with size 35                                                                                                                                                                           
[flv @ 0x15613d0] DTS 0 < 15225 out of order
[libfdk_aac @ 0x1c5c200] Queue input is backward in time                                                                                                                                                                                 
[flv @ 0x1c5a960] Non-monotonous DTS in output stream 0:1; previous: 6357, current: -8811; changing to 6357. This may result in incorrect timestamps in the output file.

[...]

[flv @ 0x1c5a960] Non-monotonous DTS in output stream 0:1; previous: 6357, current: -2582; changing to 6357. This may result in incorrect timestamps in the output file.
frame=  191 fps= 99 q=-1.0 Lsize=     323kB time=00:00:06.37 bitrate= 415.1kbits/s dup=0 drop=188    
video:271kB audio:38kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.572499%
[libx264 @ 0x1c5b1f0] frame I:7     Avg QP:12.04  size:  7908
[libx264 @ 0x1c5b1f0] frame P:184   Avg QP:16.58  size:  1205
[libx264 @ 0x1c5b1f0] mb I  I16..4: 70.8% 26.9%  2.3%
[libx264 @ 0x1c5b1f0] mb P  I16..4:  5.9%  0.6%  0.0%  P16..4:  7.4%  0.4%  0.1%  0.0%  0.0%    skip:85.5%
[libx264 @ 0x1c5b1f0] 8x8 transform intra:15.9% inter:70.5%
[libx264 @ 0x1c5b1f0] coded y,uvDC,uvAC intra: 1.9% 25.7% 6.1% inter: 0.3% 5.2% 0.4%
[libx264 @ 0x1c5b1f0] i16 v,h,dc,p: 83% 10%  1%  6%
[libx264 @ 0x1c5b1f0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 47% 15% 33%  1%  0%  1%  0%  2%  0%
[libx264 @ 0x1c5b1f0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 36% 29%  2%  1%  2%  0%  2%  1%
[libx264 @ 0x1c5b1f0] i8c dc,h,v,p: 30% 11% 49% 10%
[libx264 @ 0x1c5b1f0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x1c5b1f0] ref P L0: 58.9%  2.5% 31.3%  7.3%
[libx264 @ 0x1c5b1f0] kb/s:347.72

[FishB8]

The file was slightly too big, so I pushed it to the upload.ffmpeg.org ftp site.

File's name is "BUG-3553_DTS_Discontinuity.flv"

[Carl Eugen Hoyos]

Regression since c5ea3a00 - related to ticket #1627,
Reproducible with the following command lines:

$ ffmpeg -i BUG-3553_DTS_Discontinuity.flv -qscale 2 -strict -2 -acodec aac out.flv
$ ffmpeg -i BUG-3553_DTS_Discontinuity.flv -qscale 2 -strict -2 out.mov
$ ffmpeg -i BUG-3553_DTS_Discontinuity.flv -qscale 2 out.avi

[Michael Niedermayer]

does this issue also happen with librtmp instead of the inernal rtmp code ?
does it happen with other rtmp servers than nginx rtmp ?
ive never seen such timestamp resets in flv files before so iam not sure this file is valid
Do you know of some document that describes how such timestamp discontinuities should be handled in rtmp ?
also to quote the flv spec:

Timestamp UI24 Time in milliseconds at which the data in this tag applies.
               This value is relative to the first tag in the FLV file, which
               always has a timestamp of 0.

This does not sound to me like it ever may go backward

[FishB8]

Yes, this happens with both. I actually started with librtmp, and then switched to internal to see if it made any difference. (Off-topic: which of those 2 is recommended? which gets more dev attention?)

I have not tried it with other rtmp servers other than nginx. The timestamp reset happens when the rtmp server internally switches between source inputs. (In the example file, both sources were live) Here is a link to the blog post when the feature was introduced: ​http://nginx-rtmp.blogspot.com/2014/01/redirecting-streams-in-version-111.html

I'm not very knowledgeable of what is and isn't allowed in the rtmp spec, so if the nginx implementation is a departure from the spec, let me know and I will submit a bug report with that project. I'll ask the developer what documentation he's basing the feature off of. There is always the chance that I am not utilizing the feature correctly.

[FishB8]

Correction: That should read "flv spec" rather than "rtmp spec"

[FishB8]

I've cross-filed a bug report with the nginx rtmp server tracker. ​https://github.com/arut/nginx-rtmp-module/issues/385 Hopefully Arutyunyan can provide more insight to this than me.

[Michael Niedermayer]

Workaround added/fixed in 2fbdfba0f2d1851f894002d9d5930799629cc194