aac encoder adds extra frame(s)

[slhck]

Summary of the bug: The aac encoder adds one extra audio frame when encoding

How to reproduce:

ffmpeg -i out.mp4 -c:v copy -c:a aac out2.mp4
ffmpeg version N-78077-g5989add-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 13.100 / 55. 13.100
  libavcodec     57. 22.100 / 57. 22.100
  libavformat    57. 21.101 / 57. 21.101
  %                                                                                                                                                       !3491
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 25.100 /  6. 25.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
  Duration: 00:12:53.97, start: 0.000000, bitrate: 2180 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1857 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 317 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
[mp4 @ 0x7f8f8b842a00] Codec for stream 0 does not use global headers but container format requires global headers
Output #0, mp4, to 'out2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.21.101
    Stream #0:0(eng): Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1920x1080, q=2-31, 1857 kb/s, 25 fps, 25 tbr, 12800 tbn, 12800 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      encoder         : Lavc57.22.100 aac
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
frame=19348 fps=353 q=-1.0 Lsize=  188726kB time=00:12:53.97 bitrate=1997.5kbits/s speed=14.1x
video:175481kB audio:12669kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.306213%
[aac @ 0x7f8f8b806600] Qavg: 691.497

The input file:

ffprobe -i out.mp4 -show_streams | grep nb_frames
…
nb_frames=36280

The output file has one extra (audio) frame:

ffprobe -i out2.mp4 -show_streams | grep nb_frames
…
nb_frames=36281

This means that if you re-encode a file multiple times, audio and video will become more and more asynchronous.

The same bug affects libfdk_aac as well, although it adds two frames. When I add -shortest, I get one frame less than the original file.

[Carl Eugen Hoyos]

Regression since 151ecc2a / 89eea6df

I can reproduce the issue with the following command lines:

$ ffmpeg -f lavfi -i sine=d=1 out.aac
$ ffmpeg -i out.aac out2.aac
$ ffmpeg -i out2.aac -acodec copy -map 0 -f image2 out%2d

[Hendrik]

This is normal and expected behavior of any AAC encoder. It produces "preroll" data that the decoder is supposed to skip.

[slhck]

Replying to heleppkes:

This is normal and expected behavior of any AAC encoder. It produces "preroll" data that the decoder is supposed to skip.

How does the decoder know it's pre-roll data? Like I mentioned, if you re-encode the file multiple times it's not skipped, but added up.

[Hendrik]

The encoder sets avctx->initial_padding, and the muxers are supposed to offset the timestamps in the file appropriately, so while yes it may get longer, it should not cause sync issues.

If this is a bug, its not a bug in aacenc.

[Carl Eugen Hoyos]

How is the adts muxer supposed to set preroll in the output file?

[Hendrik]

raw files don't have timestamps, so all bets are off there.

[Hendrik]

To be precise, the encoder sets an initial_padding and produces one frame with negative timestamps. *ideally* the decoder would skip these initial samples after decoding, however unless a container has a way to signal how much padding exists... it wouldnt know how much.

[Carl Eugen Hoyos]

Isn't it a bug that ffmpeg -i input.aac output.aac makes the output file longer (and longer). Or is this a property of aac?

[Hendrik]

Its not a bug since raw aac files just don't have any way to signal how much data is to be skipped. Can't be fixed.

[slhck]

MP4 is not an advanced container then…?

Not sure how to work around this, I'm afraid.

[Hendrik]

Just adding one more audio frame doesn't really harm, because as I said it'll offset the timestamps and maintain A/V sync - and you shouldn't be re-encoding in a loop to add more and more frames in any case. ;)

[slhck]

I have a client who says they're experiencing a problem when encoding multiple times — I know it shouldn't be done for quality reasons, but the A/V sync is still lost.