Multichannel encoding with the native aac encoder crashes

[Carl Eugen Hoyos]

Regression since

commit 01344fe409da286cd377f9af610eb4c4888687ec
Author: Nathan Caldwell <​saintdev@gmail.com>
Date: Wed May 18 23:14:59 2011 -0600

aacenc: Implement dummy channel group analysis that just calls the single channel analysis for each channel.

Using samples.mplayerhq.hu/A-codecs/AC3/Canyon-5.1-48khz-448kbit.ac3 (or any 5.1 sample):

(gdb) r -i Canyon-5.1-48khz-448kbit.ac3 -strict experimental out.aac
Starting program: ffmpeg_g -i Canyon-5.1-48khz-448kbit.ac3 -strict experimental out.aac
[Thread debugging using libthread_db enabled]
ffmpeg version N-31670-g956c901, Copyright (c) 2000-2011 the FFmpeg developers
  built on Jul 28 2011 18:16:14 with gcc 4.3.2 [gcc-4_3-branch revision 141291]
  configuration:
  libavutil    51. 11. 1 / 51. 11. 1
  libavcodec   53.  9. 0 / 53.  9. 0
  libavformat  53.  6. 0 / 53.  6. 0
  libavdevice  53.  2. 0 / 53.  2. 0
  libavfilter   2. 27. 3 /  2. 27. 3
  libswscale    2.  0. 0 /  2.  0. 0
[ac3 @ 0x8cf5380] max_analyze_duration 5000000 reached at 5024000
[ac3 @ 0x8cf5380] Estimating duration from bitrate, this may be inaccurate
Input #0, ac3, from 'Canyon-5.1-48khz-448kbit.ac3':
  Duration: 00:00:37.98, start: 0.000000, bitrate: 448 kb/s
    Stream #0.0: Audio: ac3, 48000 Hz, 5.1, s16, 448 kb/s
Output #0, adts, to 'out.aac':
  Metadata:
    encoder         : Lavf53.6.0
    Stream #0.0: Audio: aac, 48000 Hz, 5.1, s16, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
psy_3gpp_analyze_channel (ctx=0x8cf18d4, channel=8, coefs=0x1, wi=0xbfffd6e8) at libavcodec/aacpsy.c:583
583                     band->energy += coefs[start+i] * coefs[start+i];
(gdb) bt
#0  psy_3gpp_analyze_channel (ctx=0x8cf18d4, channel=8, coefs=0x1, wi=0xbfffd6e8) at libavcodec/aacpsy.c:583
#1  0x085f646c in psy_3gpp_analyze (ctx=0x8cf18d4, channel=5, coeffs=0xbfffd688, wi=0xbfffd640)
    at libavcodec/aacpsy.c:751
#2  0x08550988 in aac_encode_frame (avctx=0x8cf5a80, frame=<value optimized out>, buf_size=173272,
    data=0xb7996020) at libavcodec/aacenc.c:593
#3  0x08439d33 in avcodec_encode_audio (avctx=0x8cf5a80, buf=0xb7969020 "�", buf_size=173272, samples=0x1)
    at libavcodec/utils.c:680
#4  0x0804fd37 in output_packet (ist=0x8cf9270, ist_index=0, ost_table=0x8cf08c0, nb_ostreams=1,
    pkt=0xbfffeeb4) at ffmpeg.c:1011
#5  0x08054ac5 in transcode (output_files=0x87962c0, nb_output_files=1, input_files=0x8cfb1a0,
    nb_input_files=1, stream_maps=0x0, nb_stream_maps=0) at ffmpeg.c:2819
#6  0x080558bd in main (argc=Cannot access memory at address 0x0
) at ffmpeg.c:4575
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0x85f5222 to 0x85f5262:
0x085f5222 <psy_3gpp_analyze_channel+274>:      fsts   (%edi)
0x085f5224 <psy_3gpp_analyze_channel+276>:      cmpb   $0x0,(%esi,%ebx,1)
0x085f5228 <psy_3gpp_analyze_channel+280>:      fld    %st(0)
0x085f522a <psy_3gpp_analyze_channel+282>:      fld    %st(1)
0x085f522c <psy_3gpp_analyze_channel+284>:      je     0x85f5264 <psy_3gpp_analyze_channel+340>
0x085f522e <psy_3gpp_analyze_channel+286>:      fstp   %st(0)
0x085f5230 <psy_3gpp_analyze_channel+288>:      fstp   %st(0)
0x085f5232 <psy_3gpp_analyze_channel+290>:      mov    0x70(%esp),%eax
0x085f5236 <psy_3gpp_analyze_channel+294>:      fld    %st(0)
0x085f5238 <psy_3gpp_analyze_channel+296>:      xor    %ecx,%ecx
0x085f523a <psy_3gpp_analyze_channel+298>:      lea    (%eax,%ebp,4),%edx
0x085f523d <psy_3gpp_analyze_channel+301>:      jmp    0x85f5242 <psy_3gpp_analyze_channel+306>
0x085f523f <psy_3gpp_analyze_channel+303>:      nop
0x085f5240 <psy_3gpp_analyze_channel+304>:      fxch   %st(1)
0x085f5242 <psy_3gpp_analyze_channel+306>:      flds   (%edx)
0x085f5244 <psy_3gpp_analyze_channel+308>:      add    $0x1,%ecx
0x085f5247 <psy_3gpp_analyze_channel+311>:      fmul   %st(0),%st
0x085f5249 <psy_3gpp_analyze_channel+313>:      faddp  %st,%st(2)
0x085f524b <psy_3gpp_analyze_channel+315>:      fxch   %st(1)
0x085f524d <psy_3gpp_analyze_channel+317>:      fsts   (%edi)
0x085f524f <psy_3gpp_analyze_channel+319>:      movzbl (%esi,%ebx,1),%eax
0x085f5253 <psy_3gpp_analyze_channel+323>:      flds   (%edx)
0x085f5255 <psy_3gpp_analyze_channel+325>:      add    $0x4,%edx
0x085f5258 <psy_3gpp_analyze_channel+328>:      fabs
0x085f525a <psy_3gpp_analyze_channel+330>:      fsqrt
0x085f525c <psy_3gpp_analyze_channel+332>:      cmp    %ecx,%eax
0x085f525e <psy_3gpp_analyze_channel+334>:      faddp  %st,%st(2)
0x085f5260 <psy_3gpp_analyze_channel+336>:      jg     0x85f5240 <psy_3gpp_analyze_channel+304>
End of assembler dump.
(gdb) info register
eax            0x1      1
ecx            0x0      0
edx            0x1      1
ebx            0x0      0
esp            0xbfffd060       0xbfffd060
ebp            0x0      0x0
esi            0x874f100        141881600
edi            0x8d80a80        148376192
eip            0x85f5242        0x85f5242 <psy_3gpp_analyze_channel+306>
eflags         0x10246  [ PF ZF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            66.6687469482421875      (raw 0x40058556660000000000)
st3            0.5999999940395355224609375      (raw 0x3ffe9999998000000000)
st4            3571.3314971923828125    (raw 0x400adf354dd000000000)
st5            45.60345458984375        (raw 0x4004b669f00000000000)
st6            112.2722015380859375     (raw 0x4005e08b5e0000000000)
st7            0        (raw 0x00000000000000000000)

[Danijel]

I recreated your crash, but for me it ended up in a different place. Nevertheless I think I know what might be a problem. This is the gdb dump in my case:

(gdb)  r -i Canyon-5.1-48khz-448kbit.ac3 -strict experimental out.aac
Starting program: /home/guest/ffmpeg/test/bin/ffmpeg -i Canyon-5.1-48khz-448kbit                                                                              .ac3 -strict experimental out.aac
[Thread debugging using libthread_db enabled]
ffmpeg version N-31617-g4095fa9, Copyright (c) 2000-2011 the FFmpeg developers
  built on Jul 29 2011 12:42:33 with gcc 4.6.1
  configuration: --prefix=/home/guest/ffmpeg/test --pkg-config=pkg-config --enab                                                                              le-shared --disable-static --enable-runtime-cpudetect --enable-debug=3 --disable                                                                              -optimizations --disable-stripping
  libavutil    51. 11. 0 / 51. 11. 0
  libavcodec   53.  8. 0 / 53.  8. 0
  libavformat  53.  6. 0 / 53.  6. 0
  libavdevice  53.  2. 0 / 53.  2. 0
  libavfilter   2. 27. 2 /  2. 27. 2
  libswscale    2.  0. 0 /  2.  0. 0
[ac3 @ 0x806a380] max_analyze_duration 5000000 reached at 5024000
[ac3 @ 0x806a380] Estimating duration from bitrate, this may be inaccurate
Input #0, ac3, from 'Canyon-5.1-48khz-448kbit.ac3':
  Duration: 00:00:37.98, start: 0.000000, bitrate: 448 kb/s
    Stream #0.0: Audio: ac3, 48000 Hz, 5.1, s16, 448 kb/s
File 'out.aac' already exists. Overwrite ? [y/N] y
Output #0, adts, to 'out.aac':
  Metadata:
    encoder         : Lavf53.6.0
    Stream #0.0: Audio: aac, 48000 Hz, 5.1, s16, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
0xb731384f in psy_3gpp_analyze_channel (ctx=0x80668ec, channel=6,
    coefs=0xb65e62c0, wi=0xbfffe554) at libavcodec/aacpsy.c:599
599             bands[g].thr   = FFMAX(bands[g].thr,    bands[g-1].thr * coeffs[                                                                              g].spread_hi[0]);

The code in question being:

(gdb) list 576,603
576     for (w = 0; w < wi->num_windows*16; w += 16) {
577         for (g = 0; g < num_bands; g++) {
578             AacPsyBand *band = &pch->band[w+g];
579
580             float form_factor = 0.0f;
581             band->energy = 0.0f;
582             for (i = 0; i < band_sizes[g]; i++) {
583                 band->energy += coefs[start+i] * coefs[start+i];
584                 form_factor  += sqrtf(fabs(coefs[start+i]));
585             }
586             band->thr      = band->energy * 0.001258925f;
587             band->nz_lines = form_factor / powf(band->energy / band_sizes[g], 0.25f);
588
589             start += band_sizes[g];
590         }
591     }
592     //modify thresholds and energies - spread, threshold in quiet, pre-e---Type <return> to continue, or q <return> to quit---
cho control
593     for (w = 0; w < wi->num_windows*16; w += 16) {
594         AacPsyBand *bands = &pch->band[w];
595
596         //5.4.2.3 "Spreading" & 5.4.3 "Spreaded Energy Calculation"
597         spread_en[0] = bands[0].energy;
598         for (g = 1; g < num_bands; g++) {
599             bands[g].thr   = FFMAX(bands[g].thr,    bands[g-1].thr * coeffs[g].spread_hi[0]);
600             spread_en[w+g] = FFMAX(bands[g].energy, spread_en[w+g-1] * coeffs[g].spread_hi[1]);
601         }
602         for (g = num_bands - 2; g >= 0; g--) {
603             bands[g].thr   = FFMAX(bands[g].thr,   bands[g+1].thr * coeffs[g].spread_low[0]);

My error is in line 599 and yours in 583. I will now show how I analyzed my bug and you can do the same yourself to confirm the error.

If I print the bands variable it turns out to be NULL:

(gdb) p bands
$8 = (AacPsyBand *) 0x0

Which is strange since it's being set properly to AacPsyBand *bands = &pch->band[w]; a few lines earlier and not changed anywhere else in that loop. If we print the second half of that expression we get a valid pointer:

(gdb) p &pch->band[w]
$10 = (AacPsyBand *) 0x80f1f28

Clearly the error is some sort of a memory leak causing overwriting local variables. If we look at what's really happening we can see that bands is set to a certain offset of pch->band array (&pch->band[w] is the same as pch->band+w) where the array is set to a static size of 128 as shown under this link: ​http://ffmpeg.org/doxygen/trunk/structAacPsyChannel.html

This array is then iterated from the starting offset of 96 (as shown by command p w) and increased for num_bands times which is 49. Since 96+49 is more than 128, we get a memory leak. In fact, the SEGFAULT happens way too late, since on my machine g is at value 44 when the OS realizes the error:

(gdb) p w
$12 = 96
(gdb) p num_bands
$13 = 49
(gdb) p g
$14 = 44

I guess you could try and increase the static size of the bands array to at least 596 (a bit more to make sure) in AacPsyChannel structure located in libavcodec/aacpsy.c:117, but I'm not sure if that's how the code is supposed to work. My conclusion is meerly by looking at the code. I don't really know what it does. I think there is just a bad bug in here somewhere cause the code seems to assume that num_bands is <16 and wi->num_windows is <8.

[Carl Eugen Hoyos]

Fixed by Nathan in July.