libutvideo fails to link to FFmpeg if assembly is enabled

[qyot27]

If libutvideo is compiled with assembly enabled (from ​this commit), then FFmpeg fails to link to it with undefined references to the assembly stuff:

MAN	doc/ffserver.1
TXT	doc/fate.txt
CC	cmdutils.o
CC	ffmpeg.o
LD	ffmpeg_g
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa0): undefined reference to `x86_sse2_PredictMedianAndCount_align16'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa4): undefined reference to `x86_sse2_PredictMedianAndCount_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa8): undefined reference to `x86_sse2_PredictLeftAndCount_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xac): undefined reference to `x86_sse1mmx_RestoreMedian_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb0): undefined reference to `x86_i686_HuffmanEncode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb4): undefined reference to `x86_i686_HuffmanDecode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb8): undefined reference to `x86_i686_HuffmanDecodeAndAccum'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xbc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xcc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xdc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xec): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xfc): undefined reference to `x86_sse2_ConvertULY2ToBottomupRGB24'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x100): undefined reference to `x86_sse2_ConvertULY2ToBottomupRGB32'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x104): undefined reference to `x86_sse2_ConvertULY2ToTopdownRGB24'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x108): undefined reference to `x86_sse2_ConvertULY2ToTopdownRGB32'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x10c): undefined reference to `x86_sse2_ConvertBottomupRGB24ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x110): undefined reference to `x86_sse2_ConvertBottomupRGB32ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x114): undefined reference to `x86_sse2_ConvertTopdownRGB24ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x118): undefined reference to `x86_sse2_ConvertTopdownRGB32ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x12c): undefined reference to `x86_i686_RestoreMedian_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x130): undefined reference to `x86_i686_HuffmanEncode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x134): undefined reference to `x86_i686_HuffmanDecode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x138): undefined reference to `x86_i686_HuffmanDecodeAndAccum'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x13c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x140): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x144): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x148): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x14c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x150): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x154): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x158): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x15c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x160): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x164): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x168): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x16c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x170): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x174): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x178): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Red'
collect2: ld returned 1 exit status
make: *** [ffmpeg_g] Error 1

Relevant configuration info:
$ gcc --version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

$ nasm -v
NASM version 2.09.10 compiled on Oct 17 2011

FFmpeg N-43331-gee475e2:
./configure --prefix=$HOME/ffmpeg_build --enable-gpl --enable-version3 --enable-libutvideo

[dbuitenh]

That's not official upstream, FWIW. I hacked that in. Are you sure you set the proper format for it with OS= and ARCH=? Check the format of the files NASM is producing.

[qyot27]

According to the output of file, the object files NASM produced seem to be what they're supposed to be (ELF 32-bit if I don't specify OS= and use ARCH=x86 and COFF if I make sure OS=windows and ARCH=x86).

I was working with my own ​buildsystem branch, which I had integrated that commit into. I also went back and double-checked with your's to make sure the files produced match the same way.

After some testing, if I was compiling for my 64-bit setup and enabled x64 asm, then everything linked (even with my branch), although it resulted in a segfault whenever libutvideo was used for decoding or encoding. The linking problems seem to be focused only on compiling libutvideo+asm and ffmpeg as 32-bit.

Would changing elif __i386__ to elif __x86_32__ in TunedFunc.cpp actually work? I thought about it because it would parallel the x64 pattern right above it, since that one does link (segfault notwithstanding). It seemed to let a 32-bit compile link to ffmpeg, but for all I know I just told it to ignore assembly altogether (even though running nm -s on the resultant libutvideo.a showed the *asm_x86.o files and their symbols). The 32-bit build of libutvideo+asm/ffmpeg also did not segfault the way the 64-bit one did, so I'm unsure if that's also hinting to it not actually having the assembly linked in or simply that the x64 asm has a problem (it did show some warnings while compiling, or maybe it was because I forgot to use PIC).

And while this is somewhat off-topic, the define being -DSTATIC_LIB_WITH_ASM, does that preclude the ability of shared libraries using it? I would assume it does, but I just want to be sure. I have shared compilation worked out in my branch (although I've still not hammered all the kinks out of the install step or having it build alongside the static lib*), and FFmpeg seems to work with it without problems, but I'm not sure how the STATIC_LIB_WITH_ASM thing would factor in, if at all.

*EDIT 2012-08-14 8:30 AM EST:
I've since fixed this. As far as I know, those kinks are now hammered out, and then some.

[Carl Eugen Hoyos]

Works fine here with git://github.com/qyot27/libutvideo