Filter "setpts=PTS-STARTPTS" in 2-pass may cause abnormal bitrate allocation

[Saul Baker]

Summary of the bug:

Converting an mp4 to vp9 webm displays a marked reduction in quality and small final video bitrate when the filter "setpts=PTS-STARTPTS" is applied during a 2 pass encode, 1 pass encodes do not display the same difference, nor does an arbitrary old version.

How to reproduce:

ffmpeg -y -i "source.mp4" -filter_complex "[0:v]setpts=PTS-STARTPTS[outv]" -map [outv] -pass 1 -passlogfile pass_setpts.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn -f null nul
ffmpeg -y -i "source.mp4" -filter_complex "[0:v]setpts=PTS-STARTPTS[outv]" -map [outv] -pass 2 -passlogfile pass_setpts.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn dest_2pass_setpts.webm


ffmpeg -y -i "source.mp4" -filter_complex "[0:v]null[outv]" -map [outv] -pass 1 -passlogfile pass_nofilter.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn -f null nul
ffmpeg -y -i "source.mp4" -filter_complex "[0:v]null[outv]" -map [outv] -pass 2 -passlogfile pass_nofilter.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn dest_2pass_nofilter.webm

ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_setpts.webm
ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_nofilter.webm

ffprobe output

>> ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_setpts.webm
Input #0, matroska,webm, from 'dest_2pass_setpts.webm':
  Metadata:
    COM.ANDROID.VERSION: 14
    MAJOR_BRAND     : mp42
    MINOR_VERSION   : 0
    COMPATIBLE_BRANDS: isommp42
    ENCODER         : Lavf61.5.101
  Duration: 00:00:41.57, start: 0.000000, bitrate: 65 kb/s
  Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 846x480, SAR 1:1 DAR 141:80, 30 fps, 30 tbr, 1k tbn
      Metadata:
        ENCODER         : Lavc61.11.100 libvpx-vp9
        DURATION        : 00:00:41.567000000
stream.0.time_base="1/1000"
format.size="338969"

>> ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_nofilter.webm
Input #0, matroska,webm, from 'dest_2pass_nofilter.webm':
  Metadata:
    COM.ANDROID.VERSION: 14
    MAJOR_BRAND     : mp42
    MINOR_VERSION   : 0
    COMPATIBLE_BRANDS: isommp42
    ENCODER         : Lavf61.5.101
  Duration: 00:00:41.60, start: 0.000000, bitrate: 3275 kb/s
  Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 846x480, SAR 1:1 DAR 141:80, 30 fps, 30 tbr, 1k tbn
      Metadata:
        ENCODER         : Lavc61.11.100 libvpx-vp9
        DURATION        : 00:00:41.600000000
stream.0.time_base="1/1000"
format.size="17033515"

condensed ffprobe output for 2024-08-01-git-bcf08c1171

>> ffprobe dest_2pass_setpts.webm
Duration: 00:00:41.73, start: -0.007000, bitrate: 68 kb/s

>> ffprobe dest_2pass_nofilter.webm
Duration: 00:00:41.73, start: -0.007000, bitrate: 3268 kb/s

condensed ffprobe output for 2023-01-30-git-2d202985b7

>> ffprobe dest_2pass_setpts.webm
Duration: 00:00:41.73, start: -0.007000, bitrate: 3268 kb/s

>> ffprobe dest_2pass_nofilter.webm
Duration: 00:00:41.73, start: -0.007000, bitrate: 3268 kb/s

ffmpeg version 2024-08-01-git-bcf08c1171-essentials_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13.2.0 (Rev5, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      59. 31.100 / 59. 31.100
  libavcodec     61. 11.100 / 61. 11.100
  libavformat    61.  5.101 / 61.  5.101
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.102 / 10.  2.102
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
  libpostproc    58.  2.100 / 58.  2.100

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[MasterQuestionable]

͏    Possible codec regression?

͏    "setpts=PTS-STARTPTS" may effectively do what similar to "-avoid_negative_ts 2"?
͏    https://ffmpeg.org/ffmpeg-filters.html#setpts
͏    https://trac.ffmpeg.org/wiki/Seeking#avoid_negative_ts
͏    .
͏    Alike tend to influence the codec's encoding decision.

[MasterQuestionable]

͏    Try removing excessive options to ascertain the minimal one.
͏    Notably, ͏"-map" only the video in question.

͏    See also:
͏    https://ffmpeg.org/ffmpeg-codecs.html#libvpx

[MasterQuestionable]

͏    Or probably codec irrelevant, just somehow influenced the passlog generation:
͏    https://ffmpeg.org/ffmpeg.html#Video-Options

͏    Would "-f null -" work for "-pass 1" passlog generation? [ Yes. Demonstrated in description. ]
͏    Also "-map v:0" shall be preferable over "-an" alike.
͏    (could passlog even properly handle multiple video streams..?)

[Saul Baker]

In terms of minimal reproduction, this:

ffmpeg -y -i "source.mp4" -filter_complex "[0:v]setpts=PTS-STARTPTS[outv]" -map [outv] -pass 1 -passlogfile pass.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn -f null nul
ffmpeg -y -i "source.mp4" -filter_complex "[0:v]setpts=PTS-STARTPTS[outv]" -map [outv] -pass 2 -passlogfile pass.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn dest_2pass_setpts.webm


ffmpeg -y -i "source.mp4" -filter_complex "[0:v]null[outv]" -map [outv] -pass 1 -passlogfile pass.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn -f null nul
ffmpeg -y -i "source.mp4" -filter_complex "[0:v]null[outv]" -map [outv] -pass 2 -passlogfile pass.log  -c:v libvpx-vp9 -b:v 3271553 -an -sn dest_2pass_nofilter.webm

Seems to display the same behaviour:

Input #0, matroska,webm, from 'dest_2pass_setpts.webm':
Duration: 00:00:41.57, start: 0.000000, bitrate: 65 kb/s

Input #0, matroska,webm, from 'dest_2pass_nofilter.webm':
Duration: 00:00:41.60, start: 0.000000, bitrate: 3275 kb/s

[MasterQuestionable]

͏    Edit the description directly?
͏    ​https://trac.ffmpeg.org/ticket/11126#no3

͏    Use:
͏    ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" "${In}"
͏    ; for better estimation.

͏    Probably the "pass.log" would be of interest.

[Saul Baker]

null filter log

[Saul Baker]

setpts filter pass log

[MasterQuestionable]

͏    Well... it just appears to be Base64 encoded raw data.
͏    Decoded non-Plain-Text: why even the bother..?

͏    Both Base64 strings decode to 269,784 Bytes each.
͏    Mostly of alike difference:

͏    Meanwhile I wonder why did you insist the timestamps meddling?
͏    Didn't it work without..? Or try without touching timestamps for "-pass 1"?

[Saul Baker]

Didn't it work without..? Or try without touching timestamps for "-pass 1"?

It's used deeper inside an application where arbitrary clips can be requested to be concatenated together, it finds two main uses:

Fixing the slight offsets that creep in from the multiple slicing and dicing of a segments to apply and join the results of the xfade filter.

A step improving the resolution of very short precise cuts to synchronise with audio tracks by interpolating the clips up, cutting them, joining with reset timestamps and then interpolating them back down.

Both function without it but cumulative errors creep in otherwise.

Regardless, it's often useful for inputs with timestamp weirdness.

For the logs, the diagnostic thing seems to be that fps is no longer accurate, possibly similar to ​https://trac.ffmpeg.org/ticket/11086

nofilter:

#options: 846x480 fps=30/1 timebase=1/30 bitdepth=8 cabac=1 ref=1 deblock=1:0:0 analyse=0x1:0 me=dia subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=15 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=3271 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
in:0 out:0 type:I dur:2 cpbdur:2 q:22.17 aq:16.15 tex:246309 mv:55229 misc:6102 imb:1590 pmb:0 smb:0 d:- ref:;
in:2 out:1 type:P dur:2 cpbdur:2 q:22.80 aq:16.56 tex:89046 mv:9528 misc:1114 imb:343 pmb:694 smb:553 d:- ref:0 ;
in:1 out:2 type:b dur:2 cpbdur:2 q:22.80 aq:21.46 tex:30162 mv:9949 misc:953 imb:88 pmb:607 smb:895 d:- ref:0 ;
in:5 out:3 type:P dur:2 cpbdur:2 q:22.71 aq:16.55 tex:129176 mv:11196 misc:988 imb:320 pmb:840 smb:430 d:- ref:0 ;
in:3 out:4 type:B dur:2 cpbdur:2 q:22.77 aq:21.06 tex:33786 mv:9206 misc:1080 imb:56 pmb:711 smb:823 d:- ref:0 ;



setpts:

#options: 846x480 fps=90000/1 timebase=1/90000 bitdepth=8 cabac=1 ref=1 deblock=1:0:0 analyse=0x1:0 me=dia subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=15 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=3271 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
in:0 out:0 type:I dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:129 mv:447 misc:6024 imb:1590 pmb:0 smb:0 d:- ref:;
in:2 out:1 type:P dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:0 mv:0 misc:152 imb:0 pmb:0 smb:1590 d:- ref:0 ;
in:1 out:2 type:b dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:0 mv:0 misc:152 imb:0 pmb:0 smb:1590 d:- ref:0 ;
in:5 out:3 type:P dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:0 mv:0 misc:152 imb:0 pmb:0 smb:1590 d:- ref:0 ;
in:3 out:4 type:B dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:0 mv:0 misc:160 imb:0 pmb:0 smb:1590 d:- ref:0 ;
in:4 out:5 type:b dur:6000 cpbdur:6000 q:69.00 aq:51.00 tex:0 mv:0 misc:152 imb:0 pmb:0 smb:1590 d:- ref:0 ;

[MasterQuestionable]

͏    It then makes sense...
͏    Would "-avoid_negative_ts 2" workaround?

[Saul Baker]

I tried it on the xfade+concat case and no it didn't, perhaps because it's being applied to correct timings in the filter graph rather than the outputs?

[MasterQuestionable]

͏    I have rather significant doubt on whether these things really work as intended...

[Saul Baker]

That's a rather reasonable doubt to have about any significantly complex filter graph!

But as any setpts usage seems to cause this (PTS-STARTPTS, PTS+n, PTS-n) and setpts generally finds wide use, I think it's less relevant of a doubt.

[MasterQuestionable]

͏    Perhaps.
͏    What about "setpts=PTS" (presumably no-op)..?

[Saul Baker]

Yes, and indeed a parameterless setpts as "PTS" is the default expression,

fa110c32b5168d99098dc0c50c6465054cf9d20b and f121d954ac89060cb7b07da230479cffe5bf9e5c

Mean the fps and duration stripping apply regardless of the actual expression, seemingly intended as a fix for ​https://trac.ffmpeg.org/ticket/10886.

Anton's rationale about them not being accurate after setpts application does seem reasonable, but I'm unsure:

• How it's expected to be re-applied if needed inside the filter graph without externally scripting a literal into the fps filter.
• If this combined behaviour with 2pass outweighs the perceived correctness of dropping the fps and durations rather than providing a pram do explicitly do the same.

A bit of a tension here between translating the pts under which the fps and duration are constant, and transforming them in arbitrary ways - the filter can do both and the dropping of the fps and dur only really holds for the latter.

[MasterQuestionable]

͏    Would you further confirm it's irrelevant with VP9?

͏    What about using the passlog generated without "setpts" of "-pass 1", as reference for "-pass 2"..?
͏    Or generating the 2-pass video first, process the timestamps then after?

[MasterQuestionable]

͏    Didn't quite understand your previous words.
͏    Post interpreted for reference:

[ Saul Baker @ CE 2024-08-06 13:27:57 UTC:
​https://trac.ffmpeg.org/ticket/11126#comment:17
͏    Yes, and indeed a parameterless "setpts": as "PTS" is the default expression.

͏    fa110c32b5168d99098dc0c50c6465054cf9d20b
͏    f121d954ac89060cb7b07da230479cffe5bf9e5c
͏    ; mean the FPS and duration stripping apply regardless of the actual expression:
͏    Seemingly intended as fix for: ​https://trac.ffmpeg.org/ticket/10886

͏    Anton's rationale about them not being accurate after "setpts" application, does seem reasonable; but I'm unsure:
͏    How is it expected to be re-applied if needed inside the filtergraph, without externally scripting a literal into the FPS filter?
͏    If this combined behavior with 2-pass outweighs the perceived correctness of dropping the FPS and durations, rather than providing a param to explicitly do the same?

͏    A bit of a tension here between "translating" the PTS under which the FPS and duration are constant, and "transforming" them in arbitrary ways:
͏    The filter can do both and the dropping of the FPS and duration only really holds for the latter. ]

[Saul Baker]

Would you further confirm it's irrelevant with VP9?

Not sure, libx264 looks fine visually, and shows a much smaller difference that's consistent with just doing a good job at two pass, certainly not the same bitrate crash as vp9:

> ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_setpts.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dest_2pass_setpts.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.5.101
  Duration: 00:00:41.57, start: 0.000000, bitrate: 3118 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 846x480, 3115 kb/s, 30.02 fps, 30 tbr, 90k tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.11.100 libx264
stream.0.time_base="1/90000"
format.size="16202087"

> ffprobe -hide_banner -threads 0 -show_entries "stream=time_base:format=size" -of "flat=h=0" dest_2pass_nofilter.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dest_2pass_nofilter.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.5.101
  Duration: 00:00:41.60, start: 0.000000, bitrate: 3272 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 846x480, 3269 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.11.100 libx264
stream.0.time_base="1/15360"
format.size="17017374"