Some inline asm code does not compile on Haiku 32-bit with GCC 4

[Timothy Gu]

Summary of the bug

FFmpeg does not compile on Haiku 32-bit out-of-the-box.

How to reproduce

Clone FFmpeg.

Configure with --enable-gpl (but that shouldn't matter).

Make.

CC      libavfilter/x86/vf_noise.o
libavfilter/x86/vf_noise.c: In function 'line_noise_avg_mmx':
libavfilter/x86/vf_noise.c:62:5: error: 'asm' operand has impossible constraints
     __asm__ volatile(
     ^
make: *** [libavfilter/x86/vf_noise.o] Error 1

Workarounds

1. Configure with --disable-inline-asm
2. Manually define HAVE_6REGS or HAVE_EBP_AVAILABLE to 0.
3. Manually set CFLAGS to -fno-PIC (Haiku uses PIC by default so to disable it one has to explicitly add -fno-PIC)

Interesting parts in config.log

check_ld cc
check_cc
BEGIN /tmp/ffconf.uJdXoo5E.c
    1	#include <signal.h>
    2	static void sighandler(int sig){
    3	    raise(SIGTERM);
    4	}
    5	int foo(void){
    6	    volatile int i=0;
    7	__asm__ volatile ("xorl %%ebp, %%ebp" ::: "%ebp");
    8	return i;
    9	}
   10	int (*func_ptr)(void) = foo;
   11	int main(void){
   12	    signal(SIGILL, sighandler);
   13	    signal(SIGFPE, sighandler);
   14	    signal(SIGSEGV, sighandler);
   15	#ifdef SIGBUS
   16	    signal(SIGBUS, sighandler);
   17	#endif
   18	    return func_ptr();
   19	}
END /tmp/ffconf.uJdXoo5E.c
gcc -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DPIC -std=c99 -fomit-frame-pointer -fPIC -c -o /tmp/ffconf.47o9YaJR.o /tmp/ffconf.uJdXoo5E.c
gcc -o /tmp/ffconf.L9dLDbtf /tmp/ffconf.47o9YaJR.o
check_inline_asm ebx_available ""::"b"(0)
check_cc
BEGIN /tmp/ffconf.uJdXoo5E.c
    1	void foo(void){ __asm__ volatile(""::"b"(0)); }
END /tmp/ffconf.uJdXoo5E.c
gcc -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DPIC -std=c99 -fomit-frame-pointer -fPIC -c -o /tmp/ffconf.47o9YaJR.o /tmp/ffconf.uJdXoo5E.c
/tmp/ffconf.uJdXoo5E.c: In function 'foo':
/tmp/ffconf.uJdXoo5E.c:1:17: error: inconsistent operand constraints in an 'asm'
 void foo(void){ __asm__ volatile(""::"b"(0)); }
                 ^

Compiler version

gcc (2014_12_21) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

I know.

[Michael Niedermayer]

Is there a disadvantange in fno-PIC on haiku for 32bit x86 ?
if no then nopic could be forced fr 32bit x86. I assume shared libs still work then ?

[Carl Eugen Hoyos]

Is libavfilter/x86/vf_noise.c the only failing file or are there others?
Even gcc 2.95.3 succeeds in compiling this file here on Linux without ebx (with --enable-pic).

Do the functions under HAVE_7REGS succeed compilation when you disable pic?

[Timothy Gu]

Replying to cehoyos:

Is libavfilter/x86/vf_noise.c the only failing file or are there others?

If I use make -k libpostproc/postprocess.o errors out as well. I'm not sure about others.

Even gcc 2.95.3 succeeds in compiling this file here on Linux without ebx (with --enable-pic).

In my Haiku nightly gcc 2.95.3 crashes multiple times during the compilation.

Do the functions under HAVE_7REGS succeed compilation when you disable pic?

They are not compiled as EBX is detected as not available. (Which is correct.)

[Timothy Gu]

Replying to michael:

Is there a disadvantange in fno-PIC on haiku for 32bit x86 ?

I don't know, but Haiku's official build use -fno-PIC with shared library it seems. (Not sure how that's supposed to work but…)

if no then nopic could be forced fr 32bit x86. I assume shared libs still work then ?

I don't know. I'll try.

[Timothy Gu]

Configuration	shared	static
--extra-cflags='-fno-PIC'	compiles correctly links correctly EBP available EBX available	compiles correctly EBP available EBX available
vanilla	fails to compile EBP available according to configure EBX not available

I'll have FATE running soon so that should be the final say on whether to enable -fno-PIC by default on Haiku 32-bit.

[Carl Eugen Hoyos]

Replying to Timothy_Gu:

Do the functions under HAVE_7REGS succeed compilation when you disable pic?

They are not compiled as EBX is detected as not available. (Which is correct.)

I thought EBX is not available because pic gets enabled and gets available if pic is disabled? Is that not true for Haiku?

Another question is if the test for EBP works on Haiku (it does not work for icc): How does the asm look like?

[Timothy Gu]

Replying to cehoyos:

Replying to Timothy_Gu:

Do the functions under HAVE_7REGS succeed compilation when you disable pic?

They are not compiled as EBX is detected as not available. (Which is correct.)

I thought EBX is not available because pic gets enabled and gets available if pic is disabled? Is that not true for Haiku?

It is true. Haiku enables PIC for all objects by default, including those for static libraries.

Another question is if the test for EBP works on Haiku (it does not work for icc): How does the asm look like?

Getting it right now.

[Timothy Gu]

Update: -fno-PIC does NOT fix the problem when it is used in configure, but when it is used with make. This is because when it is used in configure HAVE_7REGS is defined, which doesn't work.

For the EBP test, it does not compile with gcc:

bp cannot be used in asm here

But FFmpeg automatically adds -fomit-frame-pointer, which allows it to compile, and generate the following asm:

foo:
    pushl          %ebp
    subl      $16, %esp
    movl       $0, 12(%esp)
/APP
 # 7 "test.c" 1
    xorl     %ebp, %ebp
 # 0 "" 2
/NO_APP
    movl 12(%esp), %eax
    addl      $16, %esp
    popl           %ebp

[Carl Eugen Hoyos]

I can only guess that Haiku manages to clobber one more register apart from ebp and ebx.

[Timothy Gu]

But back to vf_noise.c. If I comment out ​this part compilation succeeds. But if I even call any function (including a noop function I created) with a variable as argument compilation fails.

A reduced test case is below:

// gcc -m32 -fPIC -fomit-frame-pointer
void noop()
{}

void line_noise_avg_mmx(int src)
{
    int var = 0;

    __asm__ volatile(
            "addl #2, %0"
            :: "r" (src+1), "r" (src+2), "r" (src+3), "r" (src+4),
               "r" (src+5), "g" (src+6)
            : "%eax"
        );

    // Doesn't compile on Haiku.
    // noop(var);
    noop(); // this compiles
}

The odd thing is that GCC on Ubuntu (4.8.2) and on haiku (4.8.4) generates the exact same assembly with noop(), but Haiku errors out on noop(var).

GCC on Ubuntu seems to add these two instructions right after the exit of inline asm:

        movl 28(%esp),  %eax   # 28(%esp) seems to be `var`
        movl    %eax , (%esp)

and Haiku GCC for some reason is simply too dumb to figure this out of everything, out.

[Michael Niedermayer]

You could disable the functions which fail to build on haiku with some #if !HAIKU, iam not sure how to detect haiku ...
or something like this could be added:

--- a/configure
+++ b/configure
@@ -5622,6 +5622,7 @@ enabled threads && ! enabled pthreads && ! enabled atomics_native && die "non pt
 if test $target_os = "haiku"; then
     disable memalign
     disable posix_memalign
+    disable ebp_available # gcc 4.8.4 (2014_12_21) on haiku at least fails if ebp is used
 fi
 
 enabled_all d3d11va d3d11_cobj CoTaskMemFree &&

I can apply this but i suspect the problem is between the compiler and the individual functions and its more a coincidence that ebp_available = 0 catches all

the "g" (-mmx_len) looks suspicous as a potential cause, the compiler here must be capable to put the intermediate negated variable in memory to avoid the need for another register

[Carl Eugen Hoyos]

Possibly related to ticket #4599.