SIG SEV in clear_blocks_sse in ff_h263_decode_mb

[Sergio Garcia]

Hi,

I am using libavcodec in my mcu project and I am latelly getting some random seg faults.

I have traced it down to be in clear_blocks_sse called by ff_h263_decode_mb.

Here is the gdb info:

(gdb) bt
#0 0x00e7fb63 in clear_blocks_sse (blocks=0xb36202e0) at libavcodec/x86/dsputil_mmx.c:539
#1 0x00c7d5e9 in ff_h263_decode_mb (s=0xb3600a60, block=0xb36202e0) at libavcodec/ituh263dec.c:634
#2 0x00bcd9af in decode_slice (s=0xb3600a60) at libavcodec/h263dec.c:215
#3 0x00bceb11 in ff_h263_decode_frame (avctx=0xb3600520, data=0xb3600940, data_size=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/h263dec.c:671
#4 0x00dea531 in avcodec_decode_video2 (avctx=0xb3600520, picture=0xb3600940, got_picture_ptr=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/utils.c:769
#5 0x080c493d in H263Decoder::DecodePacket (this=0xb36004c8, in=0xb15fbcc4 "", inLen=1393, lost=0, last=1) at /usr/local/src/mcu/media/src/h263/h263codec.cpp:476
#6 0x08077852 in VideoStream::RecVideo (this=0xb6a2ad70) at /usr/local/src/mcu/media/src/videostream.cpp:668
#7 0x08076b17 in VideoStream::startReceivingVideo (par=0xb6a2ad70) at /usr/local/src/mcu/media/src/videostream.cpp:190
#8 0x001239e9 in start_thread () from /lib/libpthread.so.0
#9 0x00662f3e in clone () from /lib/libc.so.6
(gdb) list
534 );
535 }
536
537 static void clear_blocks_sse(DCTELEM *blocks)
538 {\
539 asm volatile(
540 "xorps %%xmm0, %%xmm0 \n"
541 "mov %1, %%"REG_a" \n"
542 "1: \n"
543 "movaps %%xmm0, (%0, %%"REG_a") \n"
(gdb) print blocks
$5 = (DCTELEM *) 0xb36202e0
(gdb) print *blocks
$6 = 0
(gdb) up
#1 0x00c7d5e9 in ff_h263_decode_mb (s=0xb3600a60, block=0xb36202e0) at libavcodec/ituh263dec.c:634
634 s->dsp.clear_blocks(s->block[0]);
(gdb) print *(s->blocks)
$7 = {{0 <repeats 64 times>}, {0 <repeats 64 times>}, {0 <repeats 32 times>, 41, 41, 41, 40, 40, 40, 39, 39, 41, 41, 41, 41, 40, 40, 40, 40, 42, 41, 41, 41, 41, 40, 40,

40, 42, 42, 41, 41, 41, 40, 40, 40}, {39, 40, 40, 41, 41, 40, 40, 39, 39, 40, 40, 41, 41, 40, 40, 39, 39, 40, 40, 41, 41, 40, 40, 39, 40 <repeats 16 times>, 41, 40,
40, 39, 39, 40, 40, 41, 41, 40, 40, 39, 39, 40, 40, 41, 41, 40, 40, 39, 39, 40, 40, 41}, {130, 133, 132, 127, 127, 132, 134, 132, 130, 132, 131, 128, 128, 131, 132,
130, 131, 130, 129, 127, 128, 129, 131, 131, 133, 129, 126, 125, 126, 128, 131, 135, 132, 127, 124, 126, 128, 129, 132, 135, 129, 126, 126, 129, 132, 132, 131, 131,
129, 128, 129, 132, 134, 133, 130, 127, 132, 130, 130, 132, 134, 133, 129, 125}, {132, 131, 131, 134, 134, 131, 131, 132, 132, 131, 132, 134, 134, 132, 131, 132, 132,
131, 132, 133, 133, 132, 131, 132, 131, 131, 132, 133, 133, 132, 131, 131, 131, 132, 132, 133, 133, 132, 132, 131, 131, 132, 133, 132, 132, 133, 132, 131, 130, 132,
133, 132, 132, 133, 132, 130, 130, 133, 133, 132, 132, 133, 133, 130}, {0 <repeats 64 times>}, {0 <repeats 64 times>}}

(gdb) down
#0 0x00e7fb63 in clear_blocks_sse (blocks=0xb36202e0) at libavcodec/x86/dsputil_mmx.c:539
539 asm volatile(

Best regards
Sergio

[Carl Eugen Hoyos]

dissass and registers missing, see ​http://ffmpeg.org/bugreports.html

Is the problem reproducible with ffmpeg (the application)? Are you sure you aligned your buffers sufficiently for SSE?

[Sergio Garcia]

Hi

I am using it for a multiconference application, so it is decoding rtp video data. I could try to dump the h263 stream to a file to check if it makes ffmpeg crash. The problem ususally happen on situations with big packets losses that could cause damaged h263 streams.

I add the padding to the buffer, but it is not alligned (anyway it does not crash inmediatelly).

I have got the full info from another core dump:

(gdb) bt
#0 0x00e81882 in clear_blocks_sse (blocks=0xb36202e0) at libavcodec/x86/dsputil_mmx.c:539
#1 0x00c7ec39 in ff_h263_decode_mb (s=0xb3600a60, block=0xb36202e0) at libavcodec/ituh263dec.c:634
#2 0x00bcef1f in decode_slice (s=0xb3600a60) at libavcodec/h263dec.c:215
#3 0x00bd0059 in ff_h263_decode_frame (avctx=0xb3600520, data=0xb3600940, data_size=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/h263dec.c:671
#4 0x00dec1e1 in avcodec_decode_video2 (avctx=0xb3600520, picture=0xb3600940, got_picture_ptr=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/utils.c:772
#5 0x080c493d in H263Decoder::DecodePacket (this=0xb36004c8, in=0xb15fbcc4 "", inLen=308, lost=0, last=1) at /usr/local/src/mcu/media/src/h263/h263codec.cpp:476
#6 0x08077852 in VideoStream::RecVideo (this=0xb6a122d0) at /usr/local/src/mcu/media/src/videostream.cpp:668
#7 0x08076b17 in VideoStream::startReceivingVideo (par=0xb6a122d0) at /usr/local/src/mcu/media/src/videostream.cpp:190
#8 0x001239e9 in start_thread () from /lib/libpthread.so.0
#9 0x0066ff3e in clone () from /lib/libc.so.6
(gdb) bt
#0 0x00e81882 in clear_blocks_sse (blocks=0xb36202e0) at libavcodec/x86/dsputil_mmx.c:539
#1 0x00c7ec39 in ff_h263_decode_mb (s=0xb3600a60, block=0xb36202e0) at libavcodec/ituh263dec.c:634
#2 0x00bcef1f in decode_slice (s=0xb3600a60) at libavcodec/h263dec.c:215
#3 0x00bd0059 in ff_h263_decode_frame (avctx=0xb3600520, data=0xb3600940, data_size=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/h263dec.c:671
#4 0x00dec1e1 in avcodec_decode_video2 (avctx=0xb3600520, picture=0xb3600940, got_picture_ptr=0xb15fbc40, avpkt=0xb15fbbfc) at libavcodec/utils.c:772
#5 0x080c493d in H263Decoder::DecodePacket (this=0xb36004c8, in=0xb15fbcc4 "", inLen=308, lost=0, last=1) at /usr/local/src/mcu/media/src/h263/h263codec.cpp:476
#6 0x08077852 in VideoStream::RecVideo (this=0xb6a122d0) at /usr/local/src/mcu/media/src/videostream.cpp:668
#7 0x08076b17 in VideoStream::startReceivingVideo (par=0xb6a122d0) at /usr/local/src/mcu/media/src/videostream.cpp:190
#8 0x001239e9 in start_thread () from /lib/libpthread.so.0
#9 0x0066ff3e in clone () from /lib/libc.so.6
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0xe81862 to 0xe818a2:

0x00e81862 <clear_blocks_sse+18>: movaps %xmm0,(%edx,%eax,1)
0x00e81866 <clear_blocks_sse+22>: movaps %xmm0,0x10(%edx,%eax,1)
0x00e8186b <clear_blocks_sse+27>: movaps %xmm0,0x20(%edx,%eax,1)
0x00e81870 <clear_blocks_sse+32>: movaps %xmm0,0x30(%edx,%eax,1)
0x00e81875 <clear_blocks_sse+37>: movaps %xmm0,0x40(%edx,%eax,1)
0x00e8187a <clear_blocks_sse+42>: movaps %xmm0,0x50(%edx,%eax,1)
0x00e8187f <clear_blocks_sse+47>: movaps %xmm0,0x60(%edx,%eax,1)
0x00e81884 <clear_blocks_sse+52>: movaps %xmm0,0x70(%edx,%eax,1)
0x00e81889 <clear_blocks_sse+57>: add $0x80,%eax
0x00e8188e <clear_blocks_sse+62>: js 0xe81862 <clear_blocks_sse+18>
0x00e81890 <clear_blocks_sse+64>: ret
0x00e81891: jmp 0xe818a0 <add_bytes_mmx>
0x00e81893: nop
0x00e81894: nop
0x00e81895: nop
0x00e81896: nop
0x00e81897: nop
0x00e81898: nop
0x00e81899: nop
0x00e8189a: nop
0x00e8189b: nop
0x00e8189c: nop
0x00e8189d: nop
0x00e8189e: nop
0x00e8189f: nop
0x00e818a0 <add_bytes_mmx+0>: push %esi
0x00e818a1 <add_bytes_mmx+1>: xor %edx,%edx

End of assembler dump.
(gdb) info all-registers
eax 0x0 0
ecx 0x1170da0 18288032
edx 0xb36205e0 -1285421600
ebx 0xb3600a60 -1285551520
esp 0xb15fb94c 0xb15fb94c
ebp 0x0 0x0
esi 0x7 7
edi 0x938d 37773
eip 0xe81882 0xe81882 <clear_blocks_sse+50>
eflags 0x10286 [ PF SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
st0 -nan(0x8c8c8b8b8c8c8a8a) (raw 0xffff8c8c8b8b8c8c8a8a)
st1 -nan(0x8c008c008b008b) (raw 0xffff008c008c008b008b)
st2 -nan(0x8c8c8b8b8c8b8a8a) (raw 0xffff8c8c8b8b8c8b8a8a)
st3 -nan(0x8c008c008b008b) (raw 0xffff008c008c008b008b)
st4 -nan(0x8b008b008a008a) (raw 0xffff008b008b008a008a)
st5 -nan(0x8a008a0089008a) (raw 0xffff008a008a0089008a)
st6 -nan(0x8b008b008a008a) (raw 0xffff008b008b008a008a)
st7 -inf (raw 0xffff0000000000000000)
fctrl 0x37f 895
fstat 0x20 32
ftag 0xaaaa 43690
fiseg 0x73 115
fioff 0x80779d7 134707671
foseg 0x7b 123
fooff 0xb15fc2c4 -1319124284
fop 0x144 324
xmm0 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},

v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}

xmm1 {v4_float = {0x57000000, 0x75700000, 0x55400000, 0x55400000}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0x57, 0x56, 0x56, 0x57,

0x57, 0x57, 0x56, 0x55, 0x54, 0x55, 0x55, 0x55, 0x55, 0x55, 0x54, 0x56}, v8_int16 = {0x5657, 0x5756, 0x5757, 0x5556, 0x5554, 0x5555, 0x5555, 0x5654}, v4_int32 = {
0x57565657, 0x55565757, 0x55555554, 0x56545555}, v2_int64 = {0x5556575757565657, 0x5654555555555554}, uint128 = 0x56545555555555545556575757565657}

xmm2 {v4_float = {0x57000000, 0x65700000, 0x51540000, 0x55500000}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0x57, 0x56, 0x57, 0x57,

0x57, 0x56, 0x56, 0x55, 0x55, 0x54, 0x54, 0x54, 0x55, 0x55, 0x54, 0x55}, v8_int16 = {0x5657, 0x5757, 0x5657, 0x5556, 0x5455, 0x5454, 0x5555, 0x5554}, v4_int32 = {
0x57575657, 0x55565657, 0x54545455, 0x55545555}, v2_int64 = {0x5556565757575657, 0x5554555554545455}, uint128 = 0x55545555545454555556565757575657}

xmm3 {v4_float = {0xd5800000, 0x55600000, 0x54560000, 0x51500000}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0x56, 0x57, 0x57, 0x56,

0x56, 0x55, 0x55, 0x55, 0x56, 0x54, 0x53, 0x53, 0x54, 0x54, 0x53, 0x54}, v8_int16 = {0x5756, 0x5657, 0x5556, 0x5555, 0x5456, 0x5353, 0x5454, 0x5453}, v4_int32 = {
0x56575756, 0x55555556, 0x53535456, 0x54535454}, v2_int64 = {0x5555555656575756, 0x5453545453535456}, uint128 = 0x54535454535354565555555656575756}

xmm4 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},

v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}

xmm5 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
---Type <return> to continue, or q <return> to quit---

v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}

xmm6 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},

v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}

xmm7 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},

v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}

mxcsr 0x1f80 [ IM DM ZM OM UM PM ]
mm0 {uint64 = 0x8c8c8b8b8c8c8a8a, v2_int32 = {0x8c8c8a8a, 0x8c8c8b8b}, v4_int16 = {0x8a8a, 0x8c8c, 0x8b8b, 0x8c8c}, v8_int8 = {0x8a, 0x8a, 0x8c, 0x8c, 0x8b,

0x8b, 0x8c, 0x8c}}

mm1 {uint64 = 0x8c008c008b008b, v2_int32 = {0x8b008b, 0x8c008c}, v4_int16 = {0x8b, 0x8b, 0x8c, 0x8c}, v8_int8 = {0x8b, 0x0, 0x8b, 0x0, 0x8c, 0x0, 0x8c, 0x0}}
mm2 {uint64 = 0x8c8c8b8b8c8b8a8a, v2_int32 = {0x8c8b8a8a, 0x8c8c8b8b}, v4_int16 = {0x8a8a, 0x8c8b, 0x8b8b, 0x8c8c}, v8_int8 = {0x8a, 0x8a, 0x8b, 0x8c, 0x8b,

0x8b, 0x8c, 0x8c}}

mm3 {uint64 = 0x8c008c008b008b, v2_int32 = {0x8b008b, 0x8c008c}, v4_int16 = {0x8b, 0x8b, 0x8c, 0x8c}, v8_int8 = {0x8b, 0x0, 0x8b, 0x0, 0x8c, 0x0, 0x8c, 0x0}}
mm4 {uint64 = 0x8b008b008a008a, v2_int32 = {0x8a008a, 0x8b008b}, v4_int16 = {0x8a, 0x8a, 0x8b, 0x8b}, v8_int8 = {0x8a, 0x0, 0x8a, 0x0, 0x8b, 0x0, 0x8b, 0x0}}
mm5 {uint64 = 0x8a008a0089008a, v2_int32 = {0x89008a, 0x8a008a}, v4_int16 = {0x8a, 0x89, 0x8a, 0x8a}, v8_int8 = {0x8a, 0x0, 0x89, 0x0, 0x8a, 0x0, 0x8a, 0x0}}
mm6 {uint64 = 0x8b008b008a008a, v2_int32 = {0x8a008a, 0x8b008b}, v4_int16 = {0x8a, 0x8a, 0x8b, 0x8b}, v8_int8 = {0x8a, 0x0, 0x8a, 0x0, 0x8b, 0x0, 0x8b, 0x0}}
mm7 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
(gdb)

BR
Sergio

[Carl Eugen Hoyos]

Replying to sgarcia:

I am using it for a multiconference application, so it is decoding rtp video data. I could try to dump the h263 stream to a file to check if it makes ffmpeg crash.

Sounds like a good idea.

[reimar]

The relevant code is:
s->dsp.clear_blocks(s->block[0]);
I can't see a way for that to crash except if the pointer was corrupted or the MpegEncContext freed.
valgrind might be able to help figure it out.
However your backtrace is broken so I am rather skeptical anyway:
#0 0x00e81882 in clear_blocks_sse (blocks=0xb36202e0) at libavcodec/x86/dsputil_mmx.c:539
means the program counter is at 0x00e81882
However according to the disassembly there is no instruction starting there:
0x00e8187f <clear_blocks_sse+47>: movaps %xmm0,0x60(%edx,%eax,1)
0x00e81884 <clear_blocks_sse+52>: movaps %xmm0,0x70(%edx,%eax,1)

[Sergio Garcia]

More info..

If I run my program under valgrind I cannot reproduce the crash.

Also changing the line

s->dsp.clear_blocks(s->block[0]);

By

memset(s->block[0], 0, 2*6*64);

Also seems to fix the issue.

I run several h263/flv1 encoders while running the h263 decoder, could there be a concurrence problem with the sse assembler function?

[Carl Eugen Hoyos]

Where you able to produce a dump that allows to reproduce the problem?

[Michael Niedermayer]

Is this issue still happening?
If yes you can also try address sanitizer
but without further information or a way to reproduce this, there is nothing we can do, thus i close this as (needs more info) please reopen if this issue still happens and or you can provide more information.