libfdk_aac and aac_at encoders inflate small samples, native aac encoder doesn't

[mrapp]

Summary of the bug:

Re-encoding a 192 kb/s AAC-LC stream to 128 kb/s with either the libfdk_aac or aac_at encoders results in a stream that is 256 kb/s. The status shows ~128 kb/s until the last few seconds, then jumps up and the time index progresses slower. A trace confirms what is happening is in the last couple of seconds, there are a lot more samples of smaller size. This is probably a pathological case, but I have run across several different videos exhibiting this issue (the audio stream originates from mp4 videos). Performing a trace on a copy command from the output of each encoder shows that libfdk_aac and aac_at are inflating the size of these small samples by 100 times, whereas the native aac is preserving the original small size.

So instead of decreasing the filesize by decreasing the bitrate, the filesize was increased by 30%.

How to reproduce:

% ffmpeg -i audio.mp4 -c:a libfdk_aac -ac 2 -b:a 128k out.mp4

ffmpeg version 4.4.4 and 6.1
built on macOS 11.7.10 using MacPorts:

$ ffmpeg6 -version
ffmpeg version 6.1 Copyright (c) 2000-2023 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-1300.0.29.30)
configuration: --cc=/usr/bin/clang --datadir=/opt/local/libexec/ffmpeg6/share/data --docdir=/opt/local/libexec/ffmpeg6/share/doc --progs-suffix=6 --prefix=/opt/local/libexec/ffmpeg6 --enable-audiotoolbox --disable-indev=jack --disable-libjack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --disable-metal --disable-outdev=xv --enable-opencl --enable-sdl2 --disable-securetransport --enable-videotoolbox --disable-xlib --enable-avfilter --enable-fontconfig --enable-gnutls --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libfribidi --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libzimg --enable-libzvbi --enable-lzma --enable-shared --enable-pthreads --enable-swscale --enable-zlib --enable-libaom --enable-libsvtav1 --arch=arm64 --enable-librtmp --enable-gpl --enable-libvidstab --enable-libx264 --enable-libx265 --enable-libxvid --enable-postproc --enable-libfdk-aac --enable-nonfree
libavutil      58. 29.100 / 58. 29.100
libavcodec     60. 31.102 / 60. 31.102
libavformat    60. 16.100 / 60. 16.100
libavdevice    60.  3.100 / 60.  3.100
libavfilter     9. 12.100 /  9. 12.100
libswscale      7.  5.100 /  7.  5.100
libswresample   4. 12.100 /  4. 12.100
libpostproc    57.  3.100 / 57.  3.100

ffmpeg version 4.4.4 Copyright (c) 2000-2023 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-1300.0.29.30)
configuration: --prefix=/opt/local --cc=/usr/bin/clang --mandir=/opt/local/share/man --enable-audiotoolbox --disable-indev=jack --disable-libjack --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --enable-opencl --disable-outdev=xv --enable-sdl2 --disable-securetransport --enable-videotoolbox --enable-avfilter --enable-avresample --enable-fontconfig --enable-gnutls --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libfribidi --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libzimg --enable-libzvbi --enable-lzma --enable-pthreads --enable-shared --enable-swscale --enable-zlib --enable-libaom --enable-libsvtav1 --arch=arm64 --enable-librtmp --enable-gpl --enable-libvidstab --enable-libx264 --enable-libx265 --enable-libxvid --enable-postproc --enable-libfdk-aac --enable-nonfree
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

[mrapp]

Trace of audio conversion using libfdk_aac