hqdn3d crash (assembly)

[Cigaes]

A particular combination of pixels cause hqdn3d to crash.

How to reproduce:

$ ./ffmpeg_g -loglevel debug -s 2x4 -pix_fmt yuv420p -i /tmp/t.raw -vf hqdn3d -f null -
ffmpeg version N-44586-gb90210e Copyright (c) 2000-2012 the FFmpeg developers
  built on Sep 19 2012 12:24:19 with gcc 4.7 (Debian 4.7.1-7)
  configuration: --enable-shared --disable-static --enable-gpl --enable-libx264 --enable-libass --enable-libfreetype --assert-level=1
  libavutil      51. 73.101 / 51. 73.101
  libavcodec     54. 56.100 / 54. 56.100
  libavformat    54. 27.101 / 54. 27.101
  libavdevice    54.  2.100 / 54.  2.100
  libavfilter     3. 16.104 /  3. 16.104
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[AVIOContext @ 0x1a8caa0] Statistics: 12 bytes read, 0 seeks
Input #0, image2, from '/tmp/t.raw':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0, 1, 1/25: Video: rawvideo (I420 / 0x30323449), yuv420p, 2x4, 1/25, 25 tbr, 25 tbn, 25 tbc
[Parsed_hqdn3d_0 @ 0x1a8cd40] ls:4.000000 cs:3.000000 lt:6.000000 ct:4.500000
[buffer @ 0x1a8ea00] Setting entry with key 'video_size' to value '2x4'
[buffer @ 0x1a8ea00] Setting entry with key 'pix_fmt' to value '0'
[buffer @ 0x1a8ea00] Setting entry with key 'time_base' to value '1/25'
[buffer @ 0x1a8ea00] Setting entry with key 'pixel_aspect' to value '0/1'
[buffer @ 0x1a8ea00] Setting entry with key 'sws_param' to value 'flags=2'
[buffer @ 0x1a8ea00] Setting entry with key 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x1a8ce40] w:2 h:4 pixfmt:yuv420p tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.27.101
    Stream #0:0, 0, 1/90000: Video: rawvideo (I420 / 0x30323449), yuv420p, 2x4, 1/25, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
zsh: segmentation fault

The sample file contains:

0000000: b586 1c00 0000 3c8f 7f7f 7f7f

valgrind says:

==25957== Invalid read of size 2
==25957==    at 0x50B965E: ??? (hqdn3d.asm:103)
==25957==    by 0xE5877C7: ???
==25957==    by 0x50A2724: end_frame (vf_hqdn3d.c:115)
==25957==    by 0x50B1BC0: ff_end_frame (video.c:342)
==25957==    by 0x506759A: request_frame (buffersrc.c:379)
==25957==    by 0x5067785: av_buffersrc_add_ref (buffersrc.c:152)
==25957==    by 0x5067967: av_buffersrc_add_frame (buffersrc.c:91)
==25957==    by 0x416BF6: decode_video (ffmpeg.c:1646)
==25957==    by 0x4093E8: main (ffmpeg.c:1761)
==25957==  Address 0xffffffffee57aee0 is not stack'd, malloc'd or (recently) free'd

gdb says:

Program received signal SIGSEGV, Segmentation fault.
ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
103     HQDN3D_ROW 8
(gdb) where
#0  ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
#1  0x00000000006329c8 in ?? ()
#2  0x00002aaaaaf41725 in denoise_spatial (temporal=0x645480, spatial=0x641420, depth=8, dstride=32, sstride=<optimized out>, 
    h=4, w=2, frame_ant=0xffffffff, line_ant=0x635080, dst=<optimized out>, src=<optimized out>, hqdn3d=0x632bc0)
    at libavfilter/vf_hqdn3d.c:115
#3  denoise_depth (depth=8, temporal=0x643480, spatial=<optimized out>, dstride=32, sstride=<optimized out>, 
    h=<optimized out>, w=<optimized out>, frame_ant_ptr=<optimized out>, line_ant=0x635080, dst=<optimized out>, 
    src=<optimized out>, hqdn3d=0x632bc0) at libavfilter/vf_hqdn3d.c:153
#4  end_frame (inlink=<optimized out>) at libavfilter/vf_hqdn3d.c:338

rax            0x645480 6575232
rbx            0xffffffff       4294967295
rcx            0x6329ca 6498762
rdx            0x635082 6508674
rsi            0x636581 6514049
rdi            0x636581 6514049
rbp            0x1      0x1
rsp            0x7fffffffc940   0x7fffffffc940
r8             0x0      0
r9             0x641420 6558752
r10            0x7      7
r11            0xfffffffff0000000       -268435456
r12            0x1      1
r13            0x635080 6508672
r14            0x641420 6558752
r15            0x645480 6575232
rip            0x2aaaaaf5865e   0x2aaaaaf5865e <ff_hqdn3d_row_8_x86.loop2+52>
eflags         0x10296  [ PF AF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

The crash does not happen if assembly is disabled. The arch setting is ARCH_X86_64.

(The crash also happens with a real-world image, I just cropped very tightly.)

[Cigaes]

sample file causing the crash

[Carl Eugen Hoyos]

Fixed by Loren.