jpegls enc: crash with -vf vflip

[ami_stuff]

​http://ffmpeg.org/trac/ffmpeg/attachment/ticket/1059/599.png

(gdb) r -i 599.png -vf vflip -vcodec jpegls out.avi
Starting program: d:\mingw\msys\1.0\ffmpeg\ffmpeg_g.exe -i 599.png -vf vflip -vc
odec jpegls out.avi
[New Thread 1524.0xcb4]
ffmpeg version 0.9.1.git Copyright (c) 2000-2012 the FFmpeg developers
  built on Mar 10 2012 16:15:15 with gcc 4.6.1
  configuration: --disable-yasm --disable-ffprobe
  libavutil      51. 42.100 / 51. 42.100
  libavcodec     54. 10.100 / 54. 10.100
  libavformat    54.  2.100 / 54.  2.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 63.100 /  2. 63.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  7.100 /  0.  7.100
Input #0, image2, from '599.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgb24, 599x412, 25 tbr, 25 tbn, 25 tbc
File 'out.avi' already exists. Overwrite ? [y/N] y
w:599 h:412 pixfmt:rgb24 tb:1/1000000 sar:0/1 sws_param:
Output #0, avi, to 'out.avi':
  Metadata:
    ISFT            : Lavf54.2.100
    Stream #0:0: Video: jpegls (MJLS / 0x534C4A4D), rgb24, 599x412, q=2-31, 200
kb/s, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png -> jpegls)
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
ls_encode_line (state=<optimized out>, pb=<optimized out>, last=0x0,
    cur=0x3f8a4e0, last2=0, w=1797, stride=3, comp=0, bits=8)
    at libavcodec/jpeglsenc.c:125
125             Ra = x ? R(cur, x - stride) : R(last, x);
(gdb) bt
#0  ls_encode_line (state=<optimized out>, pb=<optimized out>, last=0x0,
    cur=0x3f8a4e0, last2=0, w=1797, stride=3, comp=0, bits=8)
    at libavcodec/jpeglsenc.c:125
#1  0x007f0302 in encode_picture_ls (avctx=0x386ab20, pkt=0x22db40,
    pict=0x22d9d8, got_packet=0x22dc0c) at libavcodec/jpeglsenc.c:326
#2  0x004f945d in avcodec_encode_video2 (avctx=0x386ab20, avpkt=0x22db40,
    frame=0x22d9d8, got_packet_ptr=0x22dc0c) at libavcodec/utils.c:1219
#3  0x00405de0 in do_video_out (s=0x386a620, ost=0x3863320,
    in_picture=0x3872ce0, ist=<optimized out>) at ffmpeg.c:1619
#4  0x00407d6c in transcode_video (pkt_pts=<optimized out>,
    got_output=<optimized out>, pkt=<optimized out>, ist=<optimized out>)
    at ffmpeg.c:2178
#5  output_packet (ist=0x3871640, ost_table=0x3863320, nb_ostreams=1,
    pkt=0x22fb28) at ffmpeg.c:2270
#6  0x0040bf3b in transcode (output_files=0x3871c80, nb_output_files=1,
    input_files=0x3871700, nb_input_files=1) at ffmpeg.c:3082
#7  0x0022ff48 in ?? ()
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb)

[Carl Eugen Hoyos]

Reproducible with ./ffmpeg -i tests/lena.pnm -vf vflip out.jls

[Stefano Sabatini]

Should be fixed in:

commit 0ca15aa066f1fad20853f5a560f13688d095ea81
Author: Stefano Sabatini <stefasab@gmail.com>
Date:   Sun Mar 18 16:42:32 2012 +0100

    lavc/jpeglsenc: fix allocation in case of negative linesize, and add malloc check
    
    Fix crash with negative linesizes, fix trac ticket #1078.