Segfault while using h264 hardware acceleration

[void]

I'm using FFmpeg integrated in mplayer (with VA-API pathes), and I'm getting a SIGSEGV while trying to watch h264 video with HW-acceleration.

I've poked around with gdb and I think this is multithreading-related. After I added mplayer option "-lavdopts threads=1", which configures libavcodec, the problem has gone.

General info

Crash happens inside a thread, in the start_frame function (libavcodec/vaapi_h264.c:234):

    vactx->slice_param_size = sizeof(VASliceParameterBufferH264);
      ^-- vactx is NULL

and vactx is AVCodecContext.hwaccel_context, which should be set to vaapi_context struct instance. But I've set a breakpoint and seen hwaccel_context being properly set at the init stage, although AVCodecContext pointer being not the same as the one in the place it crashes. I suppose it happens because AVCodecContext gets copied and its fields are replicated across worker threads (libavcodec/pthread.c), and I haven't seen hwaccel_context being copied in the code.

Considering all above, I don't think it is needed to copy hwaccel_context, but rather to force threads=1 configuration on libavcodec when using HW acceleration. Several working threads would not improve anything in the case of HW accel, would they?

Debug info

(gdb) run -vo vaapi:gl -va vaapi "<...>.mkv"
Starting program: /usr/local/mplayer-vaapi-build/mplayer -vo vaapi:gl -va vaapi "<...>.mkv"
[Thread debugging using libthread_db enabled]
MPlayer SVN-r34365-4.6.1 (C) 2000-2011 MPlayer Team

Playing <...>.mkv.
libavformat file format detected.
[matroska,webm @ 0xc798e0]max_analyze_duration 5000000 reached at 5019000
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (aac), -aid 0, -alang jpn
[lavf] stream 2: subtitle (ass), -sid 0, -slang eng
VIDEO:  [H264]  1280x720  0bpp  23.976 fps    0.0 kbps ( 0.0 kbyte/s)
Load subtitles in <...>
[vo_vaapi] Using OpenGL rendering
libva: libva version 0.32.0
Xlib:  extension "XFree86-DRI" missing on display ":0.0".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/dri/fglrx_drv_video.so
libva: va_openDriver() returns 0
Opening video filter: [ass auto=1]
Couldn't open video filter 'ass'.
ASS: cannot add video filter
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
[New Thread 0x7fffe13b6700 (LWP 12803)]
[New Thread 0x7fffe0bb5700 (LWP 12804)]
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, s16le, 0.0 kbit/0.00% (ratio: 0->192000)
Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
[AO SDL] Samplerate: 48000Hz Channels: Stereo Format s16le
[New Thread 0x7fffdc3b3700 (LWP 12805)]
[Thread 0x7fffdc3b3700 (LWP 12805) exited]
[New Thread 0x7fffdc3b3700 (LWP 12806)]
AO: [sdl] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
Unsupported PixelFormat 61
[VD_FFMPEG] Trying pixfmt=1.
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vaapi] 1280x720 => 1280x720 H.264 VA-API Acceleration 
[VD_FFMPEG] XVMC-accelerated MPEG-2.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe13b6700 (LWP 12803)]
start_frame (avctx=0x1b098c0, buffer=0x0, size=0) at libavcodec/vaapi_h264.c:234
234	    vactx->slice_param_size = sizeof(VASliceParameterBufferH264);

(gdb) bt
#0  start_frame (avctx=0x1b098c0, buffer=0x0, size=0) at libavcodec/vaapi_h264.c:234
#1  0x00000000008724bb in decode_nal_units (h=0x2504250, buf=0x1b1b780 "", buf_size=2752) at libavcodec/h264.c:3920
#2  0x0000000000872c4a in decode_frame (avctx=0x1b098c0, data=0x1e53080, data_size=0x1e53200, avpkt=<optimized out>)
    at libavcodec/h264.c:4105
#3  0x0000000000945818 in frame_worker_thread (arg=0x1e52f20) at libavcodec/pthread.c:364
#4  0x00007ffff6170efc in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff055389d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

(gdb) disass /m $pc-32,+64
Dump of assembler code from 0x9af3a7 to 0x9af3e7:
225	{
   0x00000000009af3ad <start_frame+0>:	push   %r14
   0x00000000009af3af <start_frame+2>:	push   %r13
   0x00000000009af3b1 <start_frame+4>:	push   %r12
   0x00000000009af3b3 <start_frame+6>:	push   %rbp
   0x00000000009af3b4 <start_frame+7>:	push   %rbx
   0x00000000009af3b5 <start_frame+8>:	sub    $0x10,%rsp

226	    H264Context * const h = avctx->priv_data;
   0x00000000009af3b9 <start_frame+12>:	mov    0x98(%rdi),%rbx

227	    MpegEncContext * const s = &h->s;
228	    struct vaapi_context * const vactx = avctx->hwaccel_context;
   0x00000000009af3c0 <start_frame+19>:	mov    0x3e8(%rdi),%r12

229	    VAPictureParameterBufferH264 *pic_param;
230	    VAIQMatrixBufferH264 *iq_matrix;
231	
232	    av_dlog(avctx, "start_frame()\n");
233	
234	    vactx->slice_param_size = sizeof(VASliceParameterBufferH264);
=> 0x00000000009af3c7 <start_frame+26>:	movl   $0x828,0x38(%r12)

235	
236	    /* Fill in VAPictureParameterBufferH264. */
237	    pic_param = ff_vaapi_alloc_pic_param(vactx, sizeof(VAPictureParameterBufferH264));
   0x00000000009af3d0 <start_frame+35>:	mov    $0x170,%esi
   0x00000000009af3d5 <start_frame+40>:	mov    %r12,%rdi
   0x00000000009af3d8 <start_frame+43>:	callq  0xa9f079 <ff_vaapi_alloc_pic_param>
   0x00000000009af3dd <start_frame+48>:	mov    %rax,%rbp

238	    if (!pic_param)
   0x00000000009af3e0 <start_frame+51>:	test   %rax,%rax
---Type <return> to continue, or q <return> to quit---q
Quit

(gdb) info locals
h = 0x2504250
s = 0x2504250
vactx = 0x0
pic_param = <optimized out>
iq_matrix = <optimized out>

(gdb) info registers
rax            0x10995a0	17405344
rbx            0x2504250	38814288
rcx            0x0	0
rdx            0x0	0
rsi            0x0	0
rdi            0x1b098c0	28350656
rbp            0x2504250	0x2504250
rsp            0x7fffe13b5c90	0x7fffe13b5c90
r8             0x1	1
r9             0x0	0
r10            0x0	0
r11            0x2	2
r12            0x0	0
r13            0x0	0
r14            0x0	0
r15            0x415f	16735
rip            0x9af3c7	0x9af3c7 <start_frame+26>
eflags         0x10206	[ PF IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0

[Carl Eugen Hoyos]

Please confirm that the problem is reproducible with current MPlayer svn and FFmpeg git head.

[void]

Because upstream mplayer lacks VA-API support, I use this version: ​http://gitorious.org/vaapi/mplayer, hwaccel-vaapi branch, latest available at the moment commit 5644551, dated Thu Oct 27 12:49:08 2011 +0200.

At the time of reporting, I used latest FFmpeg from ​git://git.videolan.org/ffmpeg.git, master branch, commit c25c7ba, dated Fri Dec 23 11:43:29 2011 +0100. I've also reproduced the crash using FFmpeg 32c4938 dated Sat Dec 24 12:44:30 2011 +0100.

[Michael Niedermayer]

is this still happening ?
has this been reported to the maintainer of the mplayer version used?

[Carl Eugen Hoyos]

Please reopen this ticket if you can add the missing information.