HEVC encoding (to mkv) broken since Mar 9 in ffmpeg

[John Boyle]

So, if you build ffmpeg with x265, and try to use it to encode HEVC video to an MKV file, then it dies with the following message:
Could not write header for output file #0 (incorrect codec parameters ?): Invalid data found when processing input

This is using x265 at revision 6468. Earlier versions of ffmpeg did work, but then they started dying with that message recently. I did a git-bisect; one of the commits temporarily broke the build, so I had to skip it, but I was able to narrow it down to these:

The first bad commit could be any of:
a823d0948683bd97dd58556b5740e434166209a8
38aee6408a6732b8ac86341dabf6683c08caa1c1
We cannot bisect more!

Both of these talk about HEVC, and one talks about Matroska. (There have been API changes to x265 in the past, but in this case the problem seems to be on ffmpeg's end.)

Full error output and command used:

Alvin:~ john$ ~/ffmpeg/build/ffmpeg -i Redlettermedia-TheDudebrosEpisode1TooGloseForComfort624.m4v -vcodec hevc -acodec copy out.mkv
ffmpeg version N-61313-g1f36ebf Copyright (c) 2000-2014 the FFmpeg developers
  built on Mar 11 2014 21:55:56 with gcc 4.8.2 (MacPorts gcc48 4.8.2_0)
  configuration: --enable-libx265 --enable-libopus --enable-gpl
  libavutil      52. 66.101 / 52. 66.101
  libavcodec     55. 52.102 / 55. 52.102
  libavformat    55. 34.100 / 55. 34.100
  libavdevice    55. 11.100 / 55. 11.100
  libavfilter     4.  3.100 /  4.  3.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Redlettermedia-TheDudebrosEpisode1TooGloseForComfort624.m4v':
  Metadata:
    major_brand     : M4V 
    minor_version   : 1
    compatible_brands: isomiso2avc1M4A M4V mp42
    creation_time   : 1970-01-01 00:00:00
    encoder         : Lavf52.62.0
  Duration: 00:01:13.84, start: 0.000000, bitrate: 702 kb/s
    Stream #0:0(eng): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x358 [SAR 1:1 DAR 320:179], 613 kb/s, 29.97 fps, 29.97 tbr, 2997 tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 86 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : SoundHandler
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
x265 [info]: Main profile, Level-2.1 (Main tier)
x265 [info]: WPP streams / pool / frames         : 6 / 2 / 1
x265 [info]: CU size                             : 64
x265 [info]: Max RQT depth inter / intra         : 1 / 1
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 25 / 250 / 40
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / refs          : 1 / 1 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : ABR-200 kbps / 1.0 / 1
x265 [info]: tools: rect amp rd=3 lft sao-lcu sign-hide 
Output #0, matroska, to 'out.mkv':
  Metadata:
    major_brand     : M4V 
    minor_version   : 1
    compatible_brands: isomiso2avc1M4A M4V mp42
    encoder         : Lavf55.34.100
    Stream #0:0(eng): Video: hevc (libx265), yuv420p, 640x358 [SAR 1:1 DAR 320:179], q=2-31, 200 kb/s, 90k tbn, 29.97 tbc (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac ([255][0][0][0] / 0x00FF), 48000 Hz, stereo, 86 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx265)
  Stream #0:1 -> #0:1 (copy)
Could not write header for output file #0 (incorrect codec parameters ?): Invalid data found when processing input

[John Boyle]

Forgot to mention: The above applies equally to all videos that I've tested it on. (That is, versions built before the offending commits work, and versions built after the offending commits die with the same error.)

[Hendrik]

Note that before the commits you mentioned, the generated Matroska files were invalid.

[John Boyle]

Perhaps you are right about that; I don't know the Matroska standard. However, VLC did play the files just fine.

[Carl Eugen Hoyos]

Please reopen this ticket if there is any indication that this isn't a vlc problem.

[John Boyle]

I think you must have misunderstood my words.

So, it used to be that a command like "ffmpeg -i <video> -vcodec hevc out.mkv" would cause ffmpeg to do a lot of CPU work and produce a good-size MKV file containing HEVC video.

Now, that same command, "ffmpeg -i <video> -vcodec hevc out.mkv" causes ffmpeg to die on startup with that error message, having done negligible CPU work, and produce an MKV file of size 0.

VLC could play the files produced in the first case, but I don't see why this makes it a "VLC problem". Actually, for that matter, ffmpeg could parse them too--I just tested it, it can reencode the HEVC back to mpeg4. And ffplay can play them. So, no, I'm pretty sure I can say that this isn't a VLC problem.

[Michael Niedermayer]

Fixed in dded5ed9c5eb0c3d5a953e661ea21a9019e93ea4

[John Boyle]

Confirmed working on my machine. Excellent!