Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#2310 closed defect (invalid)

x264: x264_cqm_init access beyond array causes undefined behavior in gcc48

Reported by: Safari Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords: libx264
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
can't x264 bugs be submitted with trac..? anyways...

i can get to 7, whereas quant4_mf quant4_bias0 quant4_mf dequant4_mf and unquant4_mf are defined as array [4], therefore undefined behavior happens with gcc-4.8-0.14.fc19 (infinite loop).

How to reproduce:

run x264 and it segfaults in x264_cqm_init.

small test case which causes infinite loop

#include <stdio.h>
#include <string.h>
#include <inttypes.h>
#include <stdlib.h>

uint32_t (*quant4_mf[4])[16];

int main(int argc, char *argv[])
{

int i;

for(i = 0; i < 8; i++ )
{

fprintf(stderr, "i=%d\n", i);
quant4_mf[i] = (void*)0;

}
return 0;

}

% ffmpeg -i input ... output
ffmpeg version
built on ...

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

Change History (2)

comment:1 by Carl Eugen Hoyos, 12 years ago

Keywords: libx264 added
Resolution: invalid
Status: newclosed

comment:2 by Roger Pack, 12 years ago

I think I may have just run into this on windows (cross compiling) with 4.8
Was this reported to x264-devel does anybody know?

Program received signal SIGSEGV, Segmentation fault.
0x759c79c8 in msvcrt!memcmp () from C:\Windows\syswow64\msvcrt.dll
(gdb) bt
#0 0x759c79c8 in msvcrt!memcmp () from C:\Windows\syswow64\msvcrt.dll
#1 0x004a303c in x264_cqm_init ()

Also related: http://forum.doom9.org/showthread.php?p=1623567

Version 0, edited 12 years ago by Roger Pack (next)
Note: See TracTickets for help on using tickets.