Crash when reading invalid pcx file

[Carl Eugen Hoyos]

FFmpeg crashes when reading attached broken pcx file.

(gdb) r -i crash.pcx
Starting program: ffmpeg_g -i crash.pcx
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-45121-gd067e25 Copyright (c) 2000-2012 the FFmpeg developers
  built on Oct  7 2012 04:47:57 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      51. 73.102 / 51. 73.102
  libavcodec     54. 64.100 / 54. 64.100
  libavformat    54. 29.105 / 54. 29.105
  libavdevice    54.  3.100 / 54.  3.100
  libavfilter     3. 19.102 /  3. 19.102
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 16.100 /  0. 16.100
  libpostproc    52.  1.100 / 52.  1.100

Program received signal SIGSEGV, Segmentation fault.
pcx_rle_decode (compressed=1, bytes_per_scanline=768, dst=0x15a2280 "", src=<optimized out>) at libavcodec/pcx.c:54
54                  value = *src++;
(gdb) bt
#0  pcx_rle_decode (compressed=1, bytes_per_scanline=768, dst=0x15a2280 "", src=<optimized out>) at libavcodec/pcx.c:54
#1  pcx_decode_frame (avctx=0x15a8ac0, data=0x159ff40, data_size=0x7fffffffc02c, avpkt=<optimized out>)
    at libavcodec/pcx.c:166
#2  0x000000000098a75e in avcodec_decode_video2 (avctx=0x159fb00, picture=0x159ff40,
    got_picture_ptr=got_picture_ptr@entry=0x7fffffffc02c, avpkt=avpkt@entry=0x7fffffffc060) at libavcodec/utils.c:1570
#3  0x00000000005891e4 in try_decode_frame (st=st@entry=0x1599d40, avpkt=avpkt@entry=0x15a07e0, options=0x15a01a0)
    at libavformat/utils.c:2364
#4  0x000000000058fc7e in avformat_find_stream_info (ic=0x1599280, options=0x15a01a0) at libavformat/utils.c:2740
#5  0x0000000000455b99 in opt_input_file (optctx=<optimized out>, opt=<optimized out>, filename=<optimized out>)
    at ffmpeg_opt.c:780
#6  0x00000000004630a0 in parse_option (optctx=optctx@entry=0x7fffffffcaf0, opt=0x7fffffffe2f2 "i",
    arg=0x7fffffffe2f4 "crash.pcx", options=options@entry=0xbb44a0 <options>) at cmdutils.c:320
#7  0x0000000000463478 in parse_options (optctx=optctx@entry=0x7fffffffcaf0, argc=argc@entry=3,
    argv=argv@entry=0x7fffffffde78, options=0xbb44a0 <options>, parse_arg_function=0x456820 <opt_output_file>)
    at cmdutils.c:353
#8  0x000000000044f7c0 in main (argc=3, argv=0x7fffffffde78) at ffmpeg.c:3151
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x8d5f80 to 0x8d5fc0:
   0x00000000008d5f80 <pcx_decode_frame+608>:   rex.WR sub $0xffffffffc9314500,%rax
   0x00000000008d5f86 <pcx_decode_frame+614>:   nopw   %cs:0x0(%rax,%rax,1)
   0x00000000008d5f90 <pcx_decode_frame+624>:   test   %r8d,%r8d
   0x00000000008d5f93 <pcx_decode_frame+627>:   je     0x8d61d4 <pcx_decode_frame+1204>
   0x00000000008d5f99 <pcx_decode_frame+633>:   test   %ebp,%ebp
   0x00000000008d5f9b <pcx_decode_frame+635>:   je     0x8d5fe1 <pcx_decode_frame+705>
   0x00000000008d5f9d <pcx_decode_frame+637>:   xor    %edx,%edx
   0x00000000008d5f9f <pcx_decode_frame+639>:   nop
=> 0x00000000008d5fa0 <pcx_decode_frame+640>:   movzbl (%r12),%esi
   0x00000000008d5fa5 <pcx_decode_frame+645>:   cmp    $0xbf,%sil
   0x00000000008d5fa9 <pcx_decode_frame+649>:   ja     0x8d61c0 <pcx_decode_frame+1184>
   0x00000000008d5faf <pcx_decode_frame+655>:   add    $0x1,%r12
   0x00000000008d5fb3 <pcx_decode_frame+659>:   mov    $0x1,%eax
   0x00000000008d5fb8 <pcx_decode_frame+664>:   cmp    %edx,%ebp
   0x00000000008d5fba <pcx_decode_frame+666>:   jbe    0x8d5fe1 <pcx_decode_frame+705>
   0x00000000008d5fbc <pcx_decode_frame+668>:   test   %al,%al
   0x00000000008d5fbe <pcx_decode_frame+670>:   lea    -0x1(%rax),%edi
End of assembler dump.
(gdb) info register
rax            0x263    611
rbx            0x15a2280        22684288
rcx            0x200    512
rdx            0x263    611
rsi            0x0      0
rdi            0x263    611
rbp            0x300    0x300
rsp            0x7fffffffbed0   0x7fffffffbed0
r8             0x1      1
r9             0xf4     244
r10            0x0      0
r11            0x360    864
r12            0x15d9000        22908928
r13            0x100    256
r14            0x7ffff7fbd7c0   140737353865152
r15            0x100    256
rip            0x8d5fa0 0x8d5fa0 <pcx_decode_frame+640>
eflags         0x10246  [ PF ZF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

[Carl Eugen Hoyos]

Fixed by Paul B Mahol.