OS X h264_videotoolbox encoder and codecpar API race condition with extradata when using MP4 / mov

[vade]

Summary of the bug:

On OS X, using the new send frame / receive packet / codec parameter API, it appears that there is a race condition or a lack of setting of the extra data / extra data size when specifying h264_videotoolbox encoder.

We create do our normal output context setup, make a stream, and using the new codecparam API we:

• avcodec_alloc_context3 a new AVCodecContext and set it up as per our streams setting.
• avcodec_open2 the codec
• avcodec_parameters_from_context to our target stream

We then avcodec_send_frame / avcodec_receive_packet as expected, with no errors, and mux our resulting packets using av_interleaved_write_frame.

Our resulting MP4 has invalid video samples as per Apples Quicktime Player, resulting in errors in the console :

5/27/16 12:40:50.282 PM VTDecoderXPCService[26681]: [12:40:50.282] H264VideoDecoder_DecodeFrame signalled err=-12910 (err) (createJVTLibInstance failed) at /Library/Caches/com.apple.xbs/Sources/CoreMedia_frameworks/CoreMedia-1731.15.202/Sources/VideoCodecs/H264/H264VideoDecoder.c line 2477

Quicktime Player 7, Quicklook, and other video programs also have errors opening the file.

if we switch our encoder to libx264, and everything works as expected with no errors in sight.

One can fix the above issue if on the first output packet received from avcodec_receive_packet, we set our streams codecpar extra data / extra data size from our codec context above.

This is the race condition. Pseudocode :

 ret = avcodec_send_frame(mOutputVideoEncoderContext, inputFrame);
            if( ret >= 0 ) {
                ret = avcodec_receive_packet(mOutputVideoEncoderContext, outputPacket);
                if( ret >= 0 ) {
                    encodedSuccess = 1;
                    outputPacket->stream_index = videoStream->index;
                    outputPacket->duration = mAvgVideoPacketDuration;
                    
                    // DEBUG FOR VIDEO TOOLBOX BUG
                    if( videoStream->codecpar->extradata == NULL ) {
                        videoStream->codecpar->extradata = mOutputVideoEncoderContext->extradata;
                        videoStream->codecpar->extradata_size = mOutputVideoEncoderContext->extradata_size;
                    }
                }
                else {
                    std::cout << "avcodec_receive_packet got " << av_err2str(ret) << std::endl;
                }
            }
            else {
                std::cout << "avcodec_send_frame got " << av_err2str(ret) << std::endl;
            }

This bug was verified on IRC by rkern, and I was requested to make a trac issue for official reporting of the bug.

Please note this issue is relevant to git master as of commit d93495c95411b95cfc5c050dbabfcdff1c10890a

Thank you!

[Hendrik]

This would appear to be a bug in the videotoolbox encoder, and unrelated to the new encode API.
Encoders need to set the extradata during init to work properly with formats that require global metadata. libx264 does this.

[vade]

Thanks ! Do i need to edit the bug to correctly tag video toolbox encoder as the source of the issue?

[Rick Kern]

Fixed in 98a3355c6a07a9ecf048d30b65cef22d63e8ed1a