ffmpeg fails to build on x32 ABI

[devsk]

Summary of the bug: ffmpeg-0.10.3 fails to build on x32 ABI
How to reproduce:

emerge -v ffmpeg
CC      libavfilter/libmpcodecs/vf_pullup.o
libavfilter/libmpcodecs/vf_ilpack.c: Assembler messages:
libavfilter/libmpcodecs/vf_ilpack.c:116: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:195: Error: operand type mismatch for `pop'
libavfilter/libmpcodecs/vf_ilpack.c:224: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:307: Error: operand type mismatch for `pop'
make: *** [libavfilter/libmpcodecs/vf_ilpack.o] Error 1
make: *** Waiting for unfinished jobs....
CC      libavfilter/libmpcodecs/vf_qp.o
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'config':
libavfilter/libmpcodecs/vf_mcdeint.c:199:13: warning: 'avcodec_alloc_context' is deprecated (declared at ./libavcodec/avcodec.h:4025) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c:227:13: warning: 'avcodec_open' is deprecated (declared at ./libavcodec/avcodec.h:4155) [-Wdeprecated-declarations]
 * ERROR: media-video/ffmpeg-0.10.3 failed (compile phase):
 *   emake failed

Line 116 from libavfilter/libmpcodecs/vf_ilpack.c:

#if HAVE_EBX_AVAILABLE
static void pack_li_0_MMX(unsigned char *dst, unsigned char *y,
    unsigned char *u, unsigned char *v, int w, int us, int vs)
{
    __asm__ volatile (""
        "push %%"REG_BP" \n\t"

REG_BP defined in ./libavutil/x86_cpu.h:

#if ARCH_X86_64
...
#    define REG_SP "rsp"
#    define REG_BP "rbp"
...
#elif ARCH_X86_32
...
#    define REG_SP "esp"
#    define REG_BP "ebp"
...
#else
typedef int x86_reg;

The question is: is it hard to provide support for the new x32 ABI which is now supported in kernel 3.4.x, glibc-2.16, gcc-4.7.1, binutils-2.22?

[Carl Eugen Hoyos]

Please report the results for current git head.

[devsk]

The same result with the git head:

libavfilter/libmpcodecs/vf_ilpack.c: Assembler messages:
libavfilter/libmpcodecs/vf_ilpack.c:116: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:195: Error: operand type mismatch for `pop'
libavfilter/libmpcodecs/vf_ilpack.c:224: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:307: Error: operand type mismatch for `pop'
make: *** [libavfilter/libmpcodecs/vf_ilpack.o] Error 1
make: *** Waiting for unfinished jobs....
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'filter':
libavfilter/libmpcodecs/vf_mcdeint.c:104:5: warning: 'avcodec_encode_video' is deprecated (declared at ./libavcodec/avcodec.h:4162) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'config':
libavfilter/libmpcodecs/vf_mcdeint.c:199:13: warning: 'avcodec_alloc_context' is deprecated (declared at ./libavcodec/avcodec.h:3327) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c:227:13: warning: 'avcodec_open' is deprecated (declared at ./libavcodec/avcodec.h:3461) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_pp.c: In function 'put_image':
libavfilter/libmpcodecs/vf_pp.c:150:21: warning: passing argument 1 of 'pp_postprocess' from incompatible pointer type [enabled by default]
In file included from libavfilter/libmpcodecs/vf_pp.c:36:0:
./libpostproc/postprocess.h:80:6: note: expected 'const uint8_t **' but argument is of type 'unsigned char **'
libavfilter/libmpcodecs/vf_pp.c: In function 'vf_open':
libavfilter/libmpcodecs/vf_pp.c:196:13: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
libavfilter/libmpcodecs/vf_pp.c:173:9: warning: variable 'hex_mode' set but not used [-Wunused-but-set-variable]
 * ERROR: media-video/ffmpeg-9999 failed (compile phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=media-video/ffmpeg-9999'`,
 * the complete build log and the output of `emerge -pqv '=media-video/ffmpeg-9999'`.
 * The complete build log is located at '/var/tmp/portage/media-video/ffmpeg-9999/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/media-video/ffmpeg-9999/temp/environment'.
 * Working directory: '/var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999'
 * S: '/var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999'
 * 
 * The following package has failed to build or install:
 * 
 *  (media-video/ffmpeg-9999::gentoo, ebuild scheduled for merge), Log file:
 *   '/var/tmp/portage/media-video/ffmpeg-9999/temp/build.log'

[reimar]

As a very first step, configure would need a reliable and portable way to detect x32, and I am not aware of one.
Until then you can use --disable-asm (which will basically make it rather useless speed-wise, but it's quite unlikely all the asm code will be ported any time soon, if ever).

[Elon Musk]

ilpack is no more, please reopen if this still happens.

[Carl Eugen Hoyos]

Afacit, this ticket is not related to ilpack, libmpcodecs just used to be compiled before other asm code.
The configure issue should also be fixed, x32 detection is possible with the latest toolchain iirc.

To fix this ticket, changes on the x86-64 asm code are necessary: Many changes are necessary but there is a patch that indicates this is possible and not extremely intrusive, the patch can be found in the avconv developer mailing list archive (I have no link atm).

[Carl Eugen Hoyos]

​https://lists.libav.org/pipermail/libav-devel/2013-January/042470.html
​http://thread.gmane.org/gmane.comp.video.libav.devel/41828

[Carl Eugen Hoyos]

Which commit fixed the issue?

[devsk]

Sweet Jesus! 10 years!