Invalid read in dering subfilter MMX2 code

[Clément Bœsch]

The C version looks unaffected:

☭ valgrind ./ffmpeg_g -cpuflags none -i tests/lena.pnm -vf mp=pp -f null -
==31602== Memcheck, a memory error detector
==31602== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==31602== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==31602== Command: ./ffmpeg_g -cpuflags none -i tests/lena.pnm -vf mp=pp -f null -
==31602== 
ffmpeg version N-48134-g8e09e18 Copyright (c) 2000-2012 the FFmpeg developers
  built on Dec 23 2012 16:14:18 with gcc 4.7.2 (GCC)
  configuration: --enable-gpl --enable-fontconfig --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libxvid --enable-libx264 --enable-libvpx --enable-libtheora --enable-x11grab --enable-libopenjpeg --enable-libass --enable-libmodplug --enable-libv4l2 --cc=colorgcc --samples=/home/ubitux/fate-samples --prefix=/tmp/ffinstall --enable-runtime-cpudetect --enable-libcelt
  libavutil      52. 12.100 / 52. 12.100
  libavcodec     54. 81.100 / 54. 81.100
  libavformat    54. 49.102 / 54. 49.102
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 29.101 /  3. 29.101
  libswscale      2.  1.103 /  2.  1.103
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
[Parsed_mp_0 @ 0xb042900] 'pp' is a wrapped MPlayer filter (libmpcodecs). This filter may be removed
once it has been ported to a native libavfilter.
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.49.102
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> rawvideo)
Press [q] to stop, [?] for help
[null @ 0xb03b300] Encoder did not produce proper pts, making some up.
frame=    1 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.04 bitrate=   0.0kbits/s    
video:0kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.000000%
==31602== 
==31602== HEAP SUMMARY:
==31602==     in use at exit: 0 bytes in 0 blocks
==31602==   total heap usage: 2,108 allocs, 2,108 frees, 3,145,309 bytes allocated
==31602== 
==31602== All heap blocks were freed -- no leaks are possible
==31602== 
==31602== For counts of detected and suppressed errors, rerun with: -v
==31602== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)

The MMX2 version on the other hand is:

☭ valgrind ./ffmpeg_g -cpuflags mmx2 -i tests/lena.pnm -vf mp=pp -f null -
==31603== Memcheck, a memory error detector
==31603== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==31603== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==31603== Command: ./ffmpeg_g -cpuflags mmx2 -i tests/lena.pnm -vf mp=pp -f null -
==31603== 
ffmpeg version N-48134-g8e09e18 Copyright (c) 2000-2012 the FFmpeg developers
  built on Dec 23 2012 16:14:18 with gcc 4.7.2 (GCC)
  configuration: --enable-gpl --enable-fontconfig --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libxvid --enable-libx264 --enable-libvpx --enable-libtheora --enable-x11grab --enable-libopenjpeg --enable-libass --enable-libmodplug --enable-libv4l2 --cc=colorgcc --samples=/home/ubitux/fate-samples --prefix=/tmp/ffinstall --enable-runtime-cpudetect --enable-libcelt
  libavutil      52. 12.100 / 52. 12.100
  libavcodec     54. 81.100 / 54. 81.100
  libavformat    54. 49.102 / 54. 49.102
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 29.101 /  3. 29.101
  libswscale      2.  1.103 /  2.  1.103
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
[Parsed_mp_0 @ 0xb042900] 'pp' is a wrapped MPlayer filter (libmpcodecs). This filter may be removed
once it has been ported to a native libavfilter.
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.49.102
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> rawvideo)
Press [q] to stop, [?] for help
==31603== Invalid read of size 4
==31603==    at 0xB4EDE9: dering_MMX2 (postprocess_template.c:1094)
==31603==    by 0xB5DC8C: postProcess_MMX2 (postprocess_template.c:3617)
==31603==    by 0xB73DC0: pp_postprocess (postprocess.c:615)
==31603==    by 0x4C74F9: put_image (vf_pp.c:141)
==31603==    by 0x493E01: filter_frame (vf_mp.c:826)
==31603==    by 0x49E767: default_end_frame (video.c:319)
==31603==    by 0x49F50E: ff_end_frame (video.c:341)
==31603==    by 0x47AF20: ff_filter_frame (avfilter.c:645)
==31603==    by 0x498FE3: filter_frame (vf_scale.c:413)
==31603==    by 0x49E767: default_end_frame (video.c:319)
==31603==    by 0x49F50E: ff_end_frame (video.c:341)
==31603==    by 0x47AF20: ff_filter_frame (avfilter.c:645)
==31603==  Address 0xb182e5c is 4 bytes before a block of size 6,144 alloc'd
==31603==    at 0x4C29B66: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31603==    by 0x4C29C57: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31603==    by 0xBC2FF1: av_mallocz (mem.c:92)
==31603==    by 0xB54AC8: reallocBuffers (postprocess.c:863)
==31603==    by 0xB7390B: pp_get_context (postprocess.c:923)
==31603==    by 0x4C7614: config (vf_pp.c:67)
==31603==    by 0x493CF0: config_inprops (vf_mp.c:771)
==31603==    by 0x479C4C: avfilter_config_links (avfilter.c:293)
==31603==    by 0x479C02: avfilter_config_links (avfilter.c:239)
==31603==    by 0x47C908: avfilter_graph_config (avfiltergraph.c:169)
==31603==    by 0x462551: configure_filtergraph (ffmpeg_filter.c:764)
==31603==    by 0x46AFE5: transcode_init (ffmpeg.c:2171)
==31603== 
[null @ 0xb03b300] Encoder did not produce proper pts, making some up.
frame=    1 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.04 bitrate=   0.0kbits/s    
video:0kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.000000%
==31603== 
==31603== HEAP SUMMARY:
==31603==     in use at exit: 0 bytes in 0 blocks
==31603==   total heap usage: 2,108 allocs, 2,108 frees, 3,145,309 bytes allocated
==31603== 
==31603== All heap blocks were freed -- no leaks are possible
==31603== 
==31603== For counts of detected and suppressed errors, rerun with: -v
==31603== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

[Clément Bœsch]

Note: the problem is now exactly the same with -vf pp (-vf mp=pp being dropped now).

[Carl Eugen Hoyos]

A similar problem affects SSE2.

==4768== Invalid read of size 4
==4768==    at 0x87F2998: postProcess_SSE2 (postprocess_template.c:1094)

[Clément Bœsch]

Yes that's normal: the SSE2 template is exactly the MMX2 code.