avformat_find_stream_info crashes float register

[hxuanyu]

This issue is only reproduce-able on Mac version, it's reproduce-able on 1.2.x, 1.1.x and 1.0.x branches. it works fine on 0.11.x branches.

We found this issue in our own app which uses ffmpeg libraries.
after avformat_find_stream_info is called, long double calculation get some wired result.

We used 'info float' in gdb and found that float register are filled with weird value after avformat_find_stream_info is called. I'll attach dbg screen shot for you.

I also wrote a code fragment and put it into main of ffmpeg.c,
and the executable build on Mac can reproduce the issue.

testers said it happens every time when input video format is MPEG4

the configuration used to build on Mac is : --enable-cross-compile --arch=i386 --target-os=darwin --cc=gcc-4.0 --extra-cflags='-arch i386 -mmacosx-version-min=10.2 -isysroot /Developer/SDKs/MacOSX10.4u.sdk' --extra-ldflags='-arch i386 -mmacosx-version-min=10.2 -isysroot /Developer/SDKs/MacOSX10.4u.sdk' \ --disable-mmx

the tested video will be attached, you can test it on 1.1.5 or 1.2.1 branches

[hxuanyu]

The code I used to put into main function of ffmpeg.c is

AVFormatContext *pFormatCtx = NULL;
if (avformat_open_input(&pFormatCtx, src_filename, NULL, NULL) < 0) {

fprintf(stderr, "Could not open source file %s\n", src_filename);
exit(1);

}

if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {

fprintf(stderr, "Could not find stream information\n");
exit(1);

}

long double b = 2.12;
printf("rhbc73 long double b=%Lf\n", b);
long double d = b * b;
printf("rhbc73 long double b=%Lf, d=%Lf", b, d);

and the output from Mac was

hello to play /Users/alex/Documents/sample-3gp-h263-amr.3gp
rhbc73 long double b=nan
rhbc73 long double b=2.120000, d=4.494400

the output from Windows was

hello to play c:\sample-3gp-h263-amr.3gp
rhbc73 long double b=2.120000
rhbc73 long double b=2.120000, d=4.494400

notice that some values in Mac became nan

[Carl Eugen Hoyos]

Please test current git head and if you think this is a regression, please use git bisect to find the commit introducing the problem.

[Michael Niedermayer]

Fixed in 707b2135fda3687a6eeb91411f801e437f633f30
PS: --disable-mmx is a VERY bad idea