Opened 12 years ago

Last modified 12 years ago

#2443 open enhancement

Support rx values for deshake that are not multiples of 16

Reported by: Bernardo Owned by:
Priority: wish Component: avfilter
Version: git-master Keywords: crash SIGSEGV deshake
Cc: Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

When you change the values of 'rx' or 'ry' in the deshake filter, ffmpeg/ffplay will segfault

$ ffplay -i Sam.flv -vf deshake="rx=20"

ffplay version git-2013-03-11-73b40e2 Copyright (c) 2003-2013 the FFmpeg developers
  built on Mar 31 2013 15:19:05 with gcc 4.7 (Ubuntu/Linaro 4.7.2-2ubuntu1)
  configuration: --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libspeex --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-x11grab --enable-libx264 --enable-nonfree --enable-version3 --enable-libfreetype
  libavutil      52. 19.100 / 52. 19.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 44.101 /  3. 44.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, flv, from 'Sam.flv':=    0KB vq=    0KB sq=    0B f=0/0   
  Metadata:
    starttime       : 0
    totalduration   : 21
    totaldatarate   : 1244
    bytelength      : 3322214
    canseekontime   : true
    sourcedata      : BC9111F49HH1365228066618561
    purl            : 
    pmsg            : 
  Duration: 00:00:20.80, start: 0.000000, bitrate: 1277 kb/s
    Stream #0:0: Video: h264 (Main), yuv420p, 720x480, 1163 kb/s, 30 tbr, 1k tbn, 60 tbc
    Stream #0:1: Audio: aac, 44100 Hz, stereo, fltp, 132 kb/s
Segmentation fault (core dumped)

Using ffplay_g, sometimes it will print some unrelated messages (probably invalid accesses made by the filter on other parts of the program), but most of the time it is just segfault. Some of the messages:

X Error of failed request:  BadGC (invalid GC parameter)
  Major opcode of failed request:  60 (X_FreeGC)
  Resource id in failed request:  0x6000000
  Serial number of failed request:  82
  Current serial number in output stream:  82

[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
ffplay_g: ../../src/xcb_io.c:273: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
Aborted (core dumped)

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server "�
�"
      after 82 requests (82 known processed) with 0 events remaining.

Using the default value 16 (or 32 or 48) for "rx" or "ry", the filter seems to work fine. With any other value, it will segfault, but sometimes will produce 1 or 2 frames.

With value 64, it will segfault faster -- without producing any frames. Looking at the source, there's an off-by-one writing in some 128x128 array (diff attached). I didn't dig very far for the the other segfault problem (possibly something in the "find_motion" function).

Also attatched the valgrind output.

Attachments (2)

counter.diff​ (554 bytes ) - added by Bernardo 12 years ago.
off-by-one error
valgrind.txt​ (4.4 KB ) - added by Bernardo 12 years ago.
Valgrind output for rx=4:ry=4

Download all attachments as: .zip

Change History (10)

by Bernardo, 12 years ago

Attachment: counter.diff​ added

off-by-one error

by Bernardo, 12 years ago

Attachment: valgrind.txt​ added

Valgrind output for rx=4:ry=4

comment:1 by Bernardo, 12 years ago

The input video file used is the same from ticket #1430:
​http://www.youtube.com/watch?v=627MqC6E5Yo

comment:2 by Carl Eugen Hoyos, 12 years ago

Keywords: crash SIGSEGV regression added; segfault filter removed
Priority: normal → important
Reproduced by developer: set
Status: new → open

For future tickets:

  • Please always test ffmpeg (and not ffplay): ffplay depends on an external library that is known to contain bugs. If a problem is only reproducible with ffplay (and not with ffmpeg), please mention that in the ticket description.
  • For crashes, please provide backtrace etc. as explained on ​http://ffmpeg.org/bugreports.html
  • Please upload samples to ​http://www1.datafilehost.com/ or incoming (which is currently down), your sample is 403 here. 
(gdb) r -i fate-suite/svq3/Vertical400kbit.sorenson3.mov -vf deshake=rx=20 -f null -
Starting program: ffmpeg_g -i fate-suite/svq3/Vertical400kbit.sorenson3.mov -vf deshake=rx=20 -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-51652-g38665ef Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr  7 2013 10:45:56 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack
  libavutil      52. 25.100 / 52. 25.100
  libavcodec     55.  2.100 / 55.  2.100
  libavformat    55.  1.100 / 55.  1.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 49.100 /  3. 49.100
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1608f80] max_analyze_duration 5000000 reached at 5000998 microseconds
Guessed Channel Layout for  Input Stream #0.1 : mono
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fate-suite/svq3/Vertical400kbit.sorenson3.mov':
  Metadata:
    creation_time   : 2001-03-20 16:17:18
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Sorenson Video 3
    encoder-eng     : Sorenson Video 3
  Duration: 00:00:43.58, start: 0.000000, bitrate: 580 kb/s
    Stream #0:0(eng): Video: svq3 (SVQ3 / 0x33515653), yuvj420p, 320x240, 391 kb/s, 30.02 fps, 30 tbr, 600 tbn, 600 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Audio: adpcm_ima_qt (ima4 / 0x34616D69), 44100 Hz, mono, s16p, 176 kb/s
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Output #0, null, to 'pipe:':
  Metadata:
    encoder-eng     : Sorenson Video 3
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Lavf55.1.100
    Stream #0:0(eng): Video: rawvideo (I420 / 0x30323449), yuvj420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 30 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Audio: pcm_s16le, 44100 Hz, mono, s16, 705 kb/s
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (svq3 -> rawvideo)
  Stream #0:1 -> #0:1 (adpcm_ima_qt -> pcm_s16le)
Press [q] to stop, [?] for help
Multiple frames in a packet from stream 1
[null @ 0x1609600] Encoder did not produce proper pts, making some up.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000a67f29 in sad16_sse2 (v=0x160ba60,
    blk2=0x17c7e24 "00000000,,,,,,,,000033333333666666669999666622222222", '3' <repeats 12 times>, "7777", '4' <repeats 16 times>, "2222////))))%%%%\037\037\037\037\034\034\034\034\026\026\026\026\026\026\026\026\020\020\020\020\r\r\r\r\n\n\n\n\f\f\f\f\n\n\n\n", '\a' <repeats 64 times>...,
    blk1=0x17f8978 "MMMMSSSSYYYY____eeeekkkknnnnnnnnjjjjggggaaaa[[[[UUUUTTTTQQQQUUUUTTTTWWWWZZZZ\\\\\\\\]]]]ccccffffffffjjjjllllmmmmmmmmkkkkfda_][XVNKHF>;8631.+'%\" \035\033\030\026\027\025\022\020\017\017\017\017", '\v' <repeats 20 times>, '\f' <repeats 12 times>, stride=352, h=4) at libavcodec/x86/motion_est.c:101
101         __asm__ volatile(
(gdb) bt
#0  0x0000000000a67f29 in sad16_sse2 (v=0x160ba60,
    blk2=0x17c7e24 "00000000,,,,,,,,000033333333666666669999666622222222", '3' <repeats 12 times>, "7777", '4' <repeats 16 times>, "2222////))))%%%%\037\037\037\037\034\034\034\034\026\026\026\026\026\026\026\026\020\020\020\020\r\r\r\r\n\n\n\n\f\f\f\f\n\n\n\n", '\a' <repeats 64 times>...,
    blk1=0x17f8978 "MMMMSSSSYYYY____eeeekkkknnnnnnnnjjjjggggaaaa[[[[UUUUTTTTQQQQUUUUTTTTWWWWZZZZ\\\\\\\\]]]]ccccffffffffjjjjllllmmmmmmmmkkkkfda_][XVNKHF>;8631.+'%\" \035\033\030\026\027\025\022\020\017\017\017\017", '\v' <repeats 20 times>, '\f' <repeats 12 times>, stride=352, h=4) at libavcodec/x86/motion_est.c:101
#1  0x0000000000484667 in find_block_motion (mv=<synthetic pointer>, stride=352,
    cy=112, cx=116,
    src2=0x17ed8f0 '\n' <repeats 24 times>, '\r' <repeats 12 times>, '\n' <repeats 88 times>, "\r\r\r\r", '\n' <repeats 72 times>...,
    src1=0x17be3b0 '\a' <repeats 200 times>..., deshake=0x160ba60)
    at libavfilter/vf_deshake.c:141
#2  find_motion (deshake=deshake@entry=0x160ba60,
    src1=0x17be3b0 '\a' <repeats 200 times>...,
    src2=0x17ed8f0 '\n' <repeats 24 times>, '\r' <repeats 12 times>, '\n' <repeats 88 times>, "\r\r\r\r", '\n' <repeats 72 times>..., width=320, height=240, stride=352,
    t=t@entry=0x7fffffffd320) at libavfilter/vf_deshake.c:271
#3  0x0000000000484d77 in filter_frame (link=link@entry=0x164cae0,
    in=in@entry=0x1638fe0) at libavfilter/vf_deshake.c:472
#4  0x00000000004700ea in ff_filter_frame_framed (link=link@entry=0x164cae0,
    frame=frame@entry=0x1638fe0) at libavfilter/avfilter.c:714
#5  0x0000000000471e1b in ff_filter_frame (link=link@entry=0x164cae0, frame=0x1638fe0)
    at libavfilter/avfilter.c:787
#6  0x00000000004755a2 in request_frame (link=0x164cae0)
    at libavfilter/buffersrc.c:521
#7  0x00000000004757fa in av_buffersrc_add_frame_internal (ctx=ctx@entry=0x164bae0,
    frame=frame@entry=0x1773a80, flags=flags@entry=4) at libavfilter/buffersrc.c:165
#8  0x0000000000475b4d in av_buffersrc_add_frame_flags (ctx=0x164bae0,
    frame=frame@entry=0x1773a80, flags=flags@entry=4) at libavfilter/buffersrc.c:104
#9  0x000000000045d272 in decode_video (ist=ist@entry=0x1610940,
    pkt=pkt@entry=0x7fffffffdab0, got_output=got_output@entry=0x7fffffffd84c)
    at ffmpeg.c:1714
#10 0x00000000004603ff in output_packet (pkt=0x7fffffffda50, ist=0x1610940)
    at ffmpeg.c:1832
#11 process_input (file_index=<optimized out>) at ffmpeg.c:3021
#12 0x000000000044fe00 in transcode_step () at ffmpeg.c:3117
#13 transcode () at ffmpeg.c:3169
#14 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3346
(gdb) disass $pc-25,$pc+32
Dump of assembler code from 0xa67f10 to 0xa67f49:
   0x0000000000a67f10 <sad16_sse2+0>:   movslq %ecx,%rax
   0x0000000000a67f13 <sad16_sse2+3>:   pxor   %xmm2,%xmm2
   0x0000000000a67f17 <sad16_sse2+7>:   nopw   0x0(%rax,%rax,1)
   0x0000000000a67f20 <sad16_sse2+16>:  movdqu (%rdx),%xmm0
   0x0000000000a67f24 <sad16_sse2+20>:  movdqu (%rdx,%rax,1),%xmm1
=> 0x0000000000a67f29 <sad16_sse2+25>:  psadbw (%rsi),%xmm0
   0x0000000000a67f2d <sad16_sse2+29>:  psadbw (%rsi,%rax,1),%xmm1
   0x0000000000a67f32 <sad16_sse2+34>:  paddw  %xmm0,%xmm2
   0x0000000000a67f36 <sad16_sse2+38>:  paddw  %xmm1,%xmm2
   0x0000000000a67f3a <sad16_sse2+42>:  lea    (%rdx,%rax,2),%rdx
   0x0000000000a67f3e <sad16_sse2+46>:  lea    (%rsi,%rax,2),%rsi
   0x0000000000a67f42 <sad16_sse2+50>:  sub    $0x2,%r8d
   0x0000000000a67f46 <sad16_sse2+54>:  jg     0xa67f20 <sad16_sse2+16>
   0x0000000000a67f48 <sad16_sse2+56>:  movhlps %xmm2,%xmm0
End of assembler dump.
(gdb) info all-register
rax            0x160    352
rbx            0xffffffec       4294967276
rcx            0x160    352
rdx            0x17f8978        25135480
rsi            0x17c7e24        24935972
rdi            0x160ba60        23116384
rbp            0xffffffec       0xffffffec
rsp            0x7ffffffed198   0x7ffffffed198
r8             0x4      4
r9             0x70     112
r10            0x74     116
r11            0x17ed8f0        25090288
r12            0x7fffffff       2147483647
r13            0x0      0
r14            0x160ba60        23116384
r15            0x0      0
rip            0xa67f29 0xa67f29 <sad16_sse2+25>
eflags         0x10206  [ PF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            -nan(0x8080808080808080) (raw 0xffff8080808080808080)
st1            -nan(0x8080808080808080) (raw 0xffff8080808080808080)
st2            -nan(0x002000200)        (raw 0xffff0000000002000200)
st3            -nan(0x200020002000200)  (raw 0xffff0200020002000200)
st4            -nan(0x5a5a5a545554574b) (raw 0xffff5a5a5a545554574b)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            -inf     (raw 0xffff0000000000000000)
fctrl          0x37f    895
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]
ymm0           {v8_float = {0xcd4d4d0, 0x53530000, 0x90000000, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0x4d, 0x4d, 0x4d, 0x4d, 0x53, 0x53, 0x53, 0x53, 0x59, 0x59, 0x59, 0x59, 0x5f, 0x5f, 0x5f, 0x5f, 0x0 <repeats 16 times>}, v16_int16 = {0x4d4d, 0x4d4d, 0x5353, 0x5353, 0x5959, 0x5959, 0x5f5f, 0x5f5f, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x4d4d4d4d, 0x53535353, 0x59595959, 0x5f5f5f5f, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x535353534d4d4d4d, 0x5f5f5f5f59595959, 0x0, 0x0}, v2_int128 = {0x5f5f5f5f59595959535353534d4d4d4d, 0x00000000000000000000000000000000}}
ymm1           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x8000000000000000, 0x8000000000000000, 0x0, 0x0}, v32_int8 = {0x67, 0x67, 0x67, 0x67, 0x6d, 0x6d, 0x6d, 0x6d, 0x73, 0x73, 0x73, 0x73, 0x79, 0x79, 0x79, 0x79, 0x0 <repeats 16 times>}, v16_int16 = {0x6767, 0x6767, 0x6d6d, 0x6d6d, 0x7373, 0x7373, 0x7979, 0x7979, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0x67676767, 0x6d6d6d6d, 0x73737373, 0x79797979, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x6d6d6d6d67676767, 0x7979797973737373, 0x0, 0x0}, v2_int128 = {0x79797979737373736d6d6d6d67676767, 0x00000000000000000000000000000000}}
ymm2           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm3           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xc <repeats 16 times>, 0x0 <repeats 16 times>}, v16_int16 = {0xc0c, 0xc0c, 0xc0c, 0xc0c, 0xc0c, 0xc0c, 0xc0c, 0xc0c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int32 = {0xc0c0c0c, 0xc0c0c0c, 0xc0c0c0c, 0xc0c0c0c, 0x0, 0x0, 0x0, 0x0}, v4_int64 = { 0xc0c0c0c0c0c0c0c, 0xc0c0c0c0c0c0c0c, 0x0, 0x0}, v2_int128 = {0x0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c, 0x00000000000000000000000000000000}}
ymm4           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm5           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm6           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm7           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm8           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm9           {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm10          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm11          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x80, 0x0 <repeats 24 times>}, v16_int16 = {0x0, 0x0, 0x0, 0x8000, 0x0 <repeats 12 times>}, v8_int32 = {0x0, 0x80000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x8000000000000000, 0x0, 0x0, 0x0}, v2_int128 = { 0x00000000000000008000000000000000, 0x00000000000000000000000000000000}}
ymm12          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}}
ymm13          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x80, 0x73, 0x3c, 0x0 <repeats 24 times>}, v16_int16 = {0x0, 0x0, 0x8000, 0x3c73, 0x0 <repeats 12 times>}, v8_int32 = {0x0, 0x3c738000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x3c73800000000000, 0x0, 0x0, 0x0}, v2_int128 = { 0x00000000000000003c73800000000000, 0x00000000000000000000000000000000}}
ymm14          {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = {0xdc, 0xc3, 0xe5, 0x9, 0x99, 0xd6, 0x73, 0x3c, 0x0 <repeats 24 times>}, v16_int16 = {0xc3dc, 0x9e5, 0xd699, 0x3c73, 0x0 <repeats 12 times>}, v8_int32 = {0x9e5c3dc, 0x3c73d699, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x3c73d69909e5c3dc, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000003c73d69909e5c3dc, 0x00000000000000000000000000000000}}
ymm15          {v8_float = {0x756b8000, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0xa, 0x0, 0x0, 0x0}, v32_int8 = {0x80, 0x94, 0xa, 0xcf, 0x5e, 0xcb, 0x24, 0x40, 0x0 <repeats 24 times>}, v16_int16 = {0x9480, 0xcf0a, 0xcb5e, 0x4024, 0x0 <repeats 12 times>}, v8_int32 = {0xcf0a9480, 0x4024cb5e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x4024cb5ecf0a9480, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000004024cb5ecf0a9480, 0x00000000000000000000000000000000}}

comment:3 by Carl Eugen Hoyos, 12 years ago

Maybe a regression since 34d0ea5.

comment:4 by Carl Eugen Hoyos, 12 years ago

Not a regression (and certainly not since 34d0ea5), all ry values work fine here, rx only works for 16, 32 and 48, all other values crash here, also with -vf deshake=-1:-1:-1:-1:40

comment:5 by Carl Eugen Hoyos, 12 years ago

Keywords: regression removed

comment:6 by Carl Eugen Hoyos, 12 years ago

The crash as I reproduced it does not happen with -cpuflags -sse2, -vf deshake=rx=64 triggers a (possibly) different problem even with -cpuflags 0.

Last edited 12 years ago by Carl Eugen Hoyos (previous) (diff)

in reply to:  6 comment:7 by Bernardo, 12 years ago

Replying to cehoyos:

-vf deshake=rx=64 triggers a (possibly) different problem even with -cpuflags 0.

Check the diff I sent above. There's an off-by-one in a 128x128 array. The code will access

thing[rx*2][ry*2]

comment:8 by Carl Eugen Hoyos, 12 years ago

Priority: important → wish
Summary: Segmentation fault on deshake filter → Support rx values for deshake that are not multiples of 16
Type: defect → enhancement

The crash was "fixed", this is now an enhancement request.

Note: See TracTickets for help on using tickets.