avformat_find_stream_info does not fill MPEG2-TS/HEVC resolution with HEVC decoder disabled

[Damian Dyńdo]

Summary of the bug:

I've found that if FFmpeg is built with HEVC decoder disabled (or does not have "hevc" decoder on whitelist in CLI), it will fail to acquire basic metadata (e.g. resolution, color characteristics, etc.) even though internal parsers are enabled and do parse NAL units like SPS or VUI from it (and during the call too).

It seems that in most cases it works because at the end of this logic if FFmpeg detects that the parameters are not set, it tries to decode the first sample which then correctly sets resolution and other stream metadata.

How to reproduce:

Either:

1. Build FFmpeg without HEVC decoder
2. Use FFmpeg API (even example code is fine) - avformat_find_stream_info() function on opened MPEG2-TS/HEVC file
3. Watch errors logged and check out that video stream in AVFormatContext does not have basic metadata like resolution, color space, etc. filled correctly.
4. If you enable logs on very high level, you will actually see that it does parse SPS NAL unit correctly.

or:

% ffprobe -v 50 -codec_whitelist h265 h1.ts
%
% ...
% [NULL @ 0x3ed7a528ca00] Could not find codec parameters for stream 0 (Video: hevc, 1 reference frame (HEVC / 0x43564548), none): unspecified size
% ...

I've used h1.ts file from #3487 (​http://trac.ffmpeg.org/ticket/3487) and it reproduces just as well.

---

Some analysis

I've found out that SPS NAL unit (and others) are parsed on such stack trace:

• ff_hevc_decode_nal_sps()
• hevc_parse()
• av_parser_parse2()
• parse_packet()
• read_frame_internal()
• avformat_find_stream_info()

The problem is that this information parsed there is not propagated higher (outside of the parser (AVCodecParserContext) to the codec (AVCodecContext)). This can be seen in av_parser_parse2() where there is code like:

%    if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
%        FILL(field_order);
%    }

This is one place in code I've found that does not depend on specific parser/internal data but can rewrite metadata from the parser to AVCodecContext. Adding there something like:

%    FILL(width);
%    FILL(height);

correctly propagates video resolution further. Unfortunately, this seems to be only part of missing metadata because format/pix_fmt is not correctly set/propagated either, and neither are other fields there like AVCodecContext's colorspace, color_trc or color_primaries members.

I wasn't able to find a good place to do this in a clean way. For HEVC I see that such metadata is accessible and partially set in hevc_parser.c file in hevc_parse_slice_header() function which looks currently like:

%    s->coded_width  = ps->sps->width;
%    s->coded_height = ps->sps->height;
%    s->width        = ps->sps->width  - ow->left_offset - ow->right_offset;
%    s->height       = ps->sps->height - ow->top_offset  - ow->bottom_offset;
%    s->format       = ps->sps->pix_fmt;
%    avctx->profile  = ps->sps->ptl.general_ptl.profile_idc;
%    avctx->level    = ps->sps->ptl.general_ptl.level_idc;

while it sets the parser's metadata, it is not propagated forward (only profile and level here seems to be propagated forward).

---

In the end, I think that this can affect negatively performance (fallback to decoding just to get metadata from the stream) and actually makes it impossible to obtain stream information (metadata such as resolution, color space, etc.) through parsers and avformat_find_stream_info() function and as such should have high priority for fixing it.

[Damian Dyńdo]

I've also collected logs from another built-from-scratch (this time standalone) ffmpeg/ffprobe:

~/Desktop/ffmpeg-git-20190521-amd64-static$ ./ffprobe -v 50 -codec_whitelist abcd ../h1.ts
ffprobe version N-48905-ge45e6005ce-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
  libavutil      56. 28.100 / 56. 28.100
  libavcodec     58. 52.102 / 58. 52.102
  libavformat    58. 27.103 / 58. 27.103
  libavdevice    58.  7.100 / 58.  7.100
  libavfilter     7. 53.101 /  7. 53.101
  libswscale      5.  4.101 /  5.  4.101
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Routing option codec_whitelist to both codec and muxer layer
[NULL @ 0x6e6fd00] Opening '../h1.ts' for reading
[file @ 0x6e70500] Setting default whitelist 'file,crypto'
[mpegts @ 0x6e6fd00] Format mpegts probed with size=2048 and score=50
[mpegts @ 0x6e6fd00] stream=0 stream_type=6 pid=12d prog_reg_desc=
[mpegts @ 0x6e6fd00] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
[NULL @ 0x6e747c0] Codec (hevc) not on whitelist 'abcd'
[mpegts @ 0x6e6fd00] Failed to open codec in avformat_find_stream_info
[NULL @ 0x6e747c0] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] Decoding VPS
[NULL @ 0x6e747c0] Unknown HEVC profile: 0
[NULL @ 0x6e747c0] Decoding SPS
[NULL @ 0x6e747c0] Unknown HEVC profile: 0
[NULL @ 0x6e747c0] Decoding PPS
[AVBSFContext @ 0x6ece680] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x6ece680] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x6ece680] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x6ece680] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] Codec (hevc) not on whitelist 'abcd'
[NULL @ 0x6e747c0] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x6e747c0] Decoding VPS
[NULL @ 0x6e747c0] Unknown HEVC profile: 0
[NULL @ 0x6e747c0] Decoding SPS
[NULL @ 0x6e747c0] Unknown HEVC profile: 0
[NULL @ 0x6e747c0] Decoding PPS
[NULL @ 0x6e747c0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
    Last message repeated 169 times
[mpegts @ 0x6e6fd00] max_analyze_duration 7000000 reached at 7008756 microseconds st:0
[mpegts @ 0x6e6fd00] rfps: 23.916667 0.015181
[mpegts @ 0x6e6fd00] rfps: 24.000000 0.007151
[mpegts @ 0x6e6fd00] rfps: 23.976024 0.003566
[mpegts @ 0x6e6fd00] rfps: 47.952048 0.014265
[mpegts @ 0x6e6fd00] Could not find codec parameters for stream 0 (Video: hevc, 1 reference frame (HEVC / 0x43564548), none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[mpegts @ 0x6e6fd00] After avformat_find_stream_info() pos: 0 bytes read:561932 seeks:1 frames:170
Input #0, mpegts, from '../h1.ts':
  Duration: 00:00:12.64, start: 0.080000, bitrate: 355 kb/s
  Program 1 
    Stream #0:0[0x12d], 170, 1/90000: Video: hevc, 1 reference frame (HEVC / 0x43564548), none, 23.98 tbr, 90k tbn, 90k tbc
[hevc @ 0x6eceb80] Codec (hevc) not on whitelist 'abcd'
Could not open codec for input stream 0

This shows that a lot of metadata is not parsed/propagated correctly if FFmpeg's HEVC decoder is disabled/compiled-out.

[Damian Dyńdo]

And this is an output (with resolution, etc.) if decoder is enabled:

~/Desktop/ffmpeg-git-20190521-amd64-static$ ./ffprobe -v 50 ../h1.ts
ffprobe version N-48905-ge45e6005ce-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
  libavutil      56. 28.100 / 56. 28.100
  libavcodec     58. 52.102 / 58. 52.102
  libavformat    58. 27.103 / 58. 27.103
  libavdevice    58.  7.100 / 58.  7.100
  libavfilter     7. 53.101 /  7. 53.101
  libswscale      5.  4.101 /  5.  4.101
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
[NULL @ 0x5b88c00] Opening '../h1.ts' for reading
[file @ 0x5b89500] Setting default whitelist 'file,crypto'
[mpegts @ 0x5b88c00] Format mpegts probed with size=2048 and score=50
[mpegts @ 0x5b88c00] stream=0 stream_type=6 pid=12d prog_reg_desc=
[mpegts @ 0x5b88c00] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
[hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] Decoding VPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding SPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding PPS
[AVBSFContext @ 0x5bf7e40] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x5bf7e40] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x5bf7e40] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[AVBSFContext @ 0x5bf7e40] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] Decoding VPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding SPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding PPS
[hevc @ 0x5b8d740] Format yuv420p chosen by get_format().
[hevc @ 0x5b8d740] Decoded frame with POC 0.
[hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5b8d740] Decoding VPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding SPS
[hevc @ 0x5b8d740] Unknown HEVC profile: 0
[hevc @ 0x5b8d740] Decoding PPS
[hevc @ 0x5b8d740] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
    Last message repeated 121 times
[mpegts @ 0x5b88c00] max_analyze_duration 5000000 reached at 5006256 microseconds st:0
[mpegts @ 0x5b88c00] rfps: 23.916667 0.009446
[mpegts @ 0x5b88c00] rfps: 24.000000 0.005342
[mpegts @ 0x5b88c00] rfps: 23.976024 0.003501
[mpegts @ 0x5b88c00] rfps: 47.952048 0.014003
[mpegts @ 0x5b88c00] After avformat_find_stream_info() pos: 0 bytes read:479376 seeks:2 frames:122
Input #0, mpegts, from '../h1.ts':
  Duration: 00:00:12.64, start: 0.080000, bitrate: 355 kb/s
  Program 1 
    Stream #0:0[0x12d], 122, 1/90000: Video: hevc, 1 reference frame (HEVC / 0x43564548), yuv420p(tv), 320x240 (320x256), 0/1, 23.98 tbr, 90k tbn, 90k tbc

[Carl Eugen Hoyos]

This is the expected behaviour, a change is not completely out of the question though.

[Damian Dyńdo]

Hi @cehoyos,

Thanks for the answer. Could you explain two things:

1. Why is this considered "intended behavior"? For me, it seems that demuxers and parsers should be enough to parse high-level metadata such as color-related metadata and stream resolution just from the parsers. Furthermore, this information *is* already parsed, just not passed higher (if not present from the other sources). Lastly, it is possibly a performance degradation to require to decode samples just to get information during the avformat_find_stream_info() call (when you don't really need to decode anything, the information is just there for "parsing").

2. What would be the intended way to get such metadata (having just demuxers and parsers) then?

---

What is even stranger and makes it totally non-consistent is that when I disable H264/AVC decoder (and leave only demuxer and parser) I get perfectly well metadata from avformat_find_stream_info(). This can be observed for example on this stream (​http://www.avalpa.com/assets/video/OC3.demo.ts) (randomly found on the internet) where it correctly gives me metadata for H264 tracks like:

Stream 7: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), none(progressive), 720x576 (0x0)
Stream 8: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), none(progressive), 1920x1080 (0x0)

and the color-related metadata are also correctly filled (although not printed in this log here). As such this really looks like a bug/lack of implementation from my point of view.

Best regards,
Damian Dyńdo.

[Damian Dyńdo]

Hi all,

In lieu of the above, I'm changing the type to "defect" again as this has rather high priority for us and we really believe this to be a "defect" rather then "intended behavior". Could you please take another look (especially with regard to what H264 parser does vs H265)?

Best regards,
Damian Dyńdo.