Support for HEVC with Alpha

[leemartin]

At WWDC this year, Apple introduced support for HEVC with Alpha to both their app and browser platforms. You can find their talk on the subject here:

​https://developer.apple.com/videos/play/wwdc2019/506

I'm an amateur at FFmpeg but I went down a rabbit hole today, trying to create a video in this new format using the latest version of FFmpeg. I took a rather simple approach of using ffprobe on one of their example video files (found in the resources on the link above) to get a profile and attempting to tune an ffmpeg encoding command to fit the same profile.

While the following command does recreate the profile, it does not retain the transparency of the source prores .mov file.

% ffmpeg -i puppets_with_alpha_prores.mov -c:v libx265 -tag:v hvc1 -pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 puppets_with_alpha_hevc.mov

Apple is pretty clear about how they are achieving this encoding in their talk and examples. Naturally, I didn't follow anything they said and simply tried to hack a solution.

So I'm wondering if this is something FFmpeg can do or would be able to support in the future?

Thanks for the help!

[Carl Eugen Hoyos]

A small sample file will be useful.

[leemartin]

Replying to cehoyos:

A small sample file will be useful.

Attached Apple's sample file. You can also see it in action on iOS 13 Safari with the following code block

<video src="puppets_with_alpha_hevc.mov" autoplay muted playsinline loop></video>

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

[Carl Eugen Hoyos]

Replying to leemartin:

Replying to cehoyos:

A small sample file will be useful.

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

404

[leemartin]

Sorry about that. Here's the right link:

​https://twitter.com/leemartin/status/1141336580797476865

Replying to cehoyos:

Replying to leemartin:

Replying to cehoyos:

A small sample file will be useful.

As seen in this tweet:

​https://twitter.com/leemartin/status/114133658079747686

404

[shaocaholica]

You would need to use a pix_fmt that has an alpha channel as well. Any of the yuv(a) formats thats also supported by h.265.

[leemartin]

Using my command above, I try swapping the pix_fmt for 'yuva420p' I receive the warning:

Incompatible pixel format 'yuva420p' for codec 'libx265', auto-selecting format 'yuv420p'

Is there an alpha supported pixel format you can suggest?

[shaocaholica]

I just mean if the feature is added you would need an alpha channel pix_fmt. The one you end up using will depend on whats supported as far as color sub sampling and bit depth.

[Carl Eugen Hoyos]

Note that this ticket - as shown by the sample attached - is about supporting decoding of transparency-containing (Apple) files and providing the transparency.
Encoding requests cannot be made here as FFmpeg does not contain a native hevc encoder.

[pecus]

Apple interoperability profile proposal

[pecus]

In case it might help, Apple had released a technical document that I did not see referenced here, so I uploaded it.

I really hope ffmpeg will provide support for decoding and encoding of transparency/alpha supporting video (I know that encoding depends on libx265 providing it)

[Balling]

Support in hevc_videotoolbox ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210223050227.65698-1-bouno@rouge.plala.or.jp/

Also nvidia started to add this thingy. See: #9088

As I understand x265 does not support alpha ​https://bitbucket.org/multicoreware/x265_git/issues/577/support-for-alpha-transparency-per-apple

[Alexander]

It would be great to see HEVC alpha transparency supported in FFMpeg. It might look like a minor issue but it’s really not. In WebGL toolkits like Verge3D, transparent textures are used to create various cool effects. In Chrome/Firefox/Edge we have WebM, but for Safari (for both desktop and mobile), HEVC is the only option. Right now we ​recommend our users to use FFMpeg with VideoToolbox encoder, but it is limited to Macs only. Thank you for consideration!

[Balling]

Some support ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_5F1F4DA9CFDF5521A752B8C9161A4D23A007@qq.com/

And ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_E0CDAECFC0550E2778B890163D4D39201F07@qq.com/

There is also support added in one of Nvidia's projects. ​https://github.com/nvpro-samples/vk_video_samples/issues/14

[logiclrd]

As I understand x265 does not support alpha

It does now. As of version 4.0, which dropped a month or two back, there is a --alpha command-line option. With this option, the input is assumed to be a yuva420p raw stream. As of writing this, to get this latest version you must build x265 from source. At least on my OS, the central package manager doesn't yet have version 4.0.

I have found the following sequence which seems to produce an MP4 file with HEVC with alpha content, but I am not on a Mac and have found no way to test it because the decoding support described by this ticket is not yet implemented. :-)

x265 --input-res 606x692 --fps 30000/1001 --alpha --crf 20 Zap.yuva420p Zap.hevc
mkvmerge Zap.hevc -o Zap.mkv
ffmpeg -itsscale 0.3336666666 -i Zap.mkv -c copy -vtag hvc1 Zap.mp4 -y

The -itsscale option is to fix an issue with the framerate. I couldn't find a way to get mkvmerge to produce the right output (there may be a way). By default, it seems to emit HEVC frames timestamped to 10fps. This scale factor converts from 10fps to 30000/1001 fps.

Hopefully this information is helpful to someone.

There is a tool called Zond 265 which does some fascinating inspection of internal H.265 encoding details. It does not support alpha channels yet, and I was told that the reason is that FFMPEG doesn't decode HEVC with Alpha yet.

I also found this related issue in Trac: #10901

(Does Trac have a mechanism for relationships between tickets?? A quick web search suggests that the answer is Yes, but only if you use an unmaintained plugin called TracTicketReferencePlugin...)