Interlaced HEVC Steam not Decoded Properly

[Jose Santiago]

Summary of the bug:

I have a 480i interlaced HEVC elementary stream. The HM reference decoder decodes the stream and generates 720x480 interlaced frames. FFMPEG/FFPLAY decodes the stream and 720x240 progressive frames and FFPLAY plays them at half the frame rate with the first/second field/frames bouncing up and down a line because these are actually 2 different fields of the same frame.

How to reproduce:

ffmpeg -i src13_interlaced.265 src13_interaced.yuv
ffplay src13_interlaced.265


$ ffmpeg -i src13_interlaced.265 src13_interaced.yuv
ffmpeg version git-2016-04-27-7bccbee-VF Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-16)
  configuration: --extra-version=VF --prefix=/mnt/kermit/work/git/ort/build/linux64/stage/ffmpeg --ld=/usr/bin/cc --target-os=linux --arch=x86_64 --cpu=x86_64 --enable-pic --enable-static --enable-shared --enable-rpath --enable-avfilter --enable-pthreads --enable-zlib --enable-bzlib --enable-runtime-cpudetect --enable-hardcoded-tables --disable-stripping --disable-doc --disable-dxva2 --disable-vaapi --disable-vda --disable-vdpau --yasmexe=/mnt/kermit/work/git/ort/build/linux64/tools/yasm/bin/yasm --sdl-config=/mnt/kermit/work/git/ort/build/linux64/stage/sdl/lib/../bin/sdl-config --extra-cflags='-I/mnt/kermit/work/git/ort/build/linux64/stage/zlib/include -I/mnt/kermit/work/git/ort/build/linux64/stage/bzip2/include -I/mnt/kermit/work/git/ort/build/linux64/stage/lzma/include -I/mnt/kermit/work/git/ort/build/linux64/stage/sdl/include -DVFBUILD_DISABLE_INTMATH_OPTIMIZATIONS=1' --extra-ldflags='-L/mnt/kermit/work/git/ort/build/linux64/stage/zlib/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/bzip2/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/lzma/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/sdl/lib'
  libavutil      55. 19.100 / 55. 19.100
  libavcodec     57. 28.203 / 57. 28.203
  libavformat    57. 29.200 / 57. 29.200
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 39.202 /  6. 39.202
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
VFAVUtil_MetaData_RunUnitTests() Completed Successfully.
Input #0, hevc, from 'src13_interlaced.265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 720x240, 30 fps, 30 tbr, 1200k tbn, 30 tbc
Output #0, rawvideo, to 'src13_interaced.yuv':
  Metadata:
    encoder         : Lavf57.29.200
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x240, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.28.203 rawvideo
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> rawvideo (native))
Press [q] to stop, [?] for help
frame=  336 fps=0.0 q=-0.0 size=   85050kB time=00:00:11.20 bitrate=62208.0kbitsframe=  337 fps=263 q=-0.0 size=   85303kB time=00:00:11.23 bitrate=62208.0kbitsframe=  522 fps=337 q=-0.0 Lsize=  132131kB time=00:00:17.40 bitrate=62208.0kbits/s speed=11.2x    
video:132131kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

[James]

Can you attach or upload the sample somewhere? Not much we can do otherwise

[Jose Santiago]

The files are too large to upload to the ticket. 2.5MB limit. I have provided the following files for external download:

HEVC Elementary Stream:
​http://162.97.176.4/src13_interlaced.265

Does not Play correctly with:
   ffplay src13_interlaced.265

FFMPEG Decoder Output:
​http://162.97.176.4/src13_ffmpeg.yuv

Does not Play correctly with:
   ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x480 -i src13_ffmpeg.yuv
Better but half framerate and field bounce and incorrect resolution with:
   ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x240 -i src13_ffmpeg.yuv

HM Reference Decoder Output:
​http://162.97.176.4/src13_hm.yuv

Plays correctly with:
   ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x480 -i src13_hm.yuv

[Jose Santiago]

HM Reference Decoder Test Run

[Hendrik]

Duplicate of ticket #4141

[Dave Helsley]

There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).

With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.

[Balling]

Replying to dhelsley:

There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).

With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.

Please can you look into this example ​https://yadi.sk/d/bQkXgt900uEdkg (DVB-T2 ts stream from Ostankino, USSR channel 58, Russia)? Is it the second as well? BTW, about reference decoder, is that this one? ​https://sourceforge.net/p/mjpeg/mailman/message/9927377/ or what?

[Dave Helsley]

Replying to Balling:

Replying to dhelsley:

There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).

With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.

Please can you look into this example ​https://yadi.sk/d/bQkXgt900uEdkg (DVB-T2 ts stream from Ostankino, USSR channel 58, Russia)? Is it the second as well? BTW, about reference decoder, is that this one? ​https://sourceforge.net/p/mjpeg/mailman/message/9927377/ or what?

Wow, old bug.

Your stream has field_seq_flag=1 and pic_struct=10,11 which is similar to our case (pic_struct=1,2 is top/bottom field, pic_struct=10,11 is top with next bottom/bottom with previous top or Top Field First).

Reference software can be found here ​https://hevc.hhi.fraunhofer.de/

[Balling]

Replying to dhelsley:

pic_struct=10,11 is top with next bottom/bottom

I am afraid it is more difficult than that? If you play in Potplayer it will oscillate between i and ib (at least there are no other crazy types ​https://sourceforge.net/p/mjpeg/mailman/message/9927377/)

Fraunhofer, of course, LOL. He is everywhere, even VoLTE EVS (enhanced voice service) or VVC/H.266.

[Balling]

Well, it was partially implemented in 876dada0b58b5ecc80b2a25eb5c33974a71c8eb2 and
7c8b65f688ea75496e278b7c042f2eda746f3eac

So when you do ffprobe src13_interlaced_cut.265 it says

Input #0, hevc, from 'C:\Users\ZAQU\Downloads\src13_interlaced_cut.265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv, top first), 720x240, 30 fps, 30 tbr, 1200k tbn, 30 t

But with ​https://yadi.sk/d/bQkXgt900uEdkg it is not like that, it says it is progressive, that is incorrect. I think I should try -filter:v idet. (P.S. Idet does not work even on src13_interlaced_cut.265)

Also after 876dada0b58b5ecc80b2a25eb5c33974a71c8eb2 it is showing itself in ffprobe -show_frames but even there there are not enough parameters as it is not just

interlaced_frame=1
top_field_first=1

[Aman]

I have a simple patch to make ffprobe report these hevc files as interlaced. Not sure if it is completely correct though, as the spec was hard for me to parse. See page 326 of Rec. ITU-T H.265 v7 (11/2019)

diff --git a/libavcodec/hevc_parser.c b/libavcodec/hevc_parser.c
index b444b99955..da9b9019e2 100644
--- a/libavcodec/hevc_parser.c
+++ b/libavcodec/hevc_parser.c
@@ -65,7 +65,24 @@ static int hevc_parse_slice_header(AVCodecParserContext *s, H2645NAL *nal,

     sh->first_slice_in_pic_flag = get_bits1(gb);
     s->picture_structure = sei->picture_timing.picture_struct;
-    s->field_order = sei->picture_timing.picture_struct;
+    switch (s->picture_structure) {
+        case 0:
+            s->field_order = AV_FIELD_PROGRESSIVE;
+            break;
+        case 1:
+        case 9:
+        case 11:
+            s->field_order = AV_FIELD_TT;
+            break;
+        case 2:
+        case 10:
+        case 12:
+            s->field_order = AV_FIELD_BB;
+            break;
+        default:
+            s->field_order = AV_FIELD_UNKNOWN;
+            break;
+    }

     if (IS_IRAP_NAL(nal)) {
         s->key_frame = 1;

[Aman]

This makes ffprobe report the correct height, but does not help with playback at all.

diff --git a/libavcodec/hevc_parser.c b/libavcodec/hevc_parser.c
index b444b99955..d00093d592 100644
--- a/libavcodec/hevc_parser.c
+++ b/libavcodec/hevc_parser.c
@@ -105,6 +105,10 @@ static int hevc_parse_slice_header(AVCodecParserContext *s, H2645NAL *nal,
         den = ps->sps->vui.vui_time_scale;
     }

+    if (ps->sps->ptl.general_ptl.interlaced_source_flag) {
+        avctx->height = s->height * 2;
+    }
+
     if (num != 0 && den != 0)
         av_reduce(&avctx->framerate.den, &avctx->framerate.num,
                   num, den, 1 << 30);

[Balling]

I think that interlaced_source_flag is not intended for check here, better check field_seq_flag for 1 as per spec ("field_seq_flag equal to 1 indicates that the CVS conveys pictures that represent fields" and further) and that is how it is done in ​https://gitlab.com/mbunkus/mkvtoolnix/-/commit/47058a6da2672944aa4544f8ecb7b9a79ba3f752#0dd3dcd183ebb1497b9a9e3522e4d11a174384a2_282_288

That will not fix #4141 of course, but that file there is not spec compliant. It can further be fixed by checking for picture_struct and interlaced_source_flag.

Indeed as I read the spec (and as mkver reads it), the fields (not frames! ) can be progressive in all cases, not only for frame doubling and trippling.

[Balling]

Workaround, works for pic_struct=10,11:

ffplay "test" -vcodec hevc_cuvid -vf weave,hwupload_cuda,yadif_cuda=mode=0:parity=0:deint=0,hwdownload,format=nv12

You may want to -bsf:v hevc_mp4toannexb it first.

[kasper93]

I don't know why there are two tickets about essentially the same issue. So I will bump this one too, with fresh sample of DVB-T2 HEVC broadcast ​https://www.mediafire.com/file/dm849hm4ykqyzvj Should stay alive for a long time, as I see few samples are not available anymore.

[Jose Santiago]

I have submitted a patch to support interlaced decoding in HEVCDEC. It seems works on all of the streams i have access too atm. https://ffmpeg.org//pipermail/ffmpeg-devel/2024-October/335529.html

[Balling]

Your patch did not apply properly:

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/6bc8f78c-ec30-4697-845a-086cff73e51f@haivision.com/

[MasterQuestionable]

͏    Actually you may just push to:
͏    ​https://github.com/FFmpeg/FFmpeg
͏    ; and share the commit link for review...
͏    (seems everyone can make dangling commits there)

͏    See also: ​https://trac.ffmpeg.org/ticket/11194#comment:5

[Balling]

It literally says that merge requests are not allowed ​https://github.com/FFmpeg/FFmpeg/pulls

[MasterQuestionable]

͏    Not creating PR.
͏    But push directly. (`git` CLI, or ​͏"github.dev")

͏    It regardless serves as a good way for review.

͏    ----

͏    ​https://github.com/FFmpeg/FFmpeg/commit/861baf8bb785d2d70d926554116ca909fa74adc5
͏    ​https://github.com/FFmpeg/FFmpeg/compare/861baf8bb785d2d70d926554116ca909fa74adc5...1d4036ac85cff781b907eaef85acb38b09bc0c7c
͏    ​https://github.com/jlsantiago0/FFmpeg/commits/hevcdec-interlaced?since=2024-10-31

[Jose Santiago]

OK. I forked the GITHUB project and created a dev branch in my GITHUB repo:

​https://github.com/jlsantiago0/FFmpeg/tree/hevcdec-interlaced

This branch has my patch.

[Jose Santiago]

I fixed a memory leak. The latest code can be found ​​https://github.com/jlsantiago0/FFmpeg/tree/hevcdec-interlaced

[Balling]

How does it compare with nvidia hack in comment 14?

[Jose Santiago]

Replying to Balling:

How does it compare with nvidia hack in comment 14?

Dont know.