H.264 decoder is left in broken state after a glitch and switch in input

[jeeb]

Summary of the bug:
After what seems to be a glitch/switch in the input stream (trace_headers notes that there's 12 H.264 packets for which the BSF couldn't be applied to), the H.264 decoder still keeps decoding, but ultimately is left in a state where it seems to drop references even though it wouldn't have to. Additionally, it seems like the stream switches from MBAFF to PAFF after the glitch, although I'm not 100% sure if I'm parsing things correctly to note that.

A cut 10MiB sample is available @ ​https://kuroko.fushizen.eu/samples/2018-08-17-h264_decoder_borks.cut.ts

How to reproduce:
You can decode or play the sample with pretty much anything lavc-using. It will be OK at first for a moment, and then have a glitch. Then after that glitch there's seemingly OK content (if you initialize decoding after the glitch, the decoding is OK), but the result is borked for the rest of the clip.

% ffmpeg -v debug -i 2018-08-17-h264_decoder_borks.cut.ts -map 0:v:0 -f null -t 10 -
... decodes fine for a while ...
timestamp discontinuity
...
[h264 @ 0x3aacbc0] reference count overflow
[h264 @ 0x3aacbc0] decode_slice_header error
[h264 @ 0x3aacbc0] no frame!
...
[h264 @ 0x3a47080] Reference 2 >= 2
[h264 @ 0x3a47080] error while decoding MB 1 0, bytestream 18674
[h264 @ 0x3bdc740] number of reference frames (0+5) exceeds max (4; probably corrupt input), discarding one
...
[h264 @ 0x3a218c0] illegal short term buffer state detected
[h264 @ 0x3a218c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x3a218c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x3a218c0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 3
[h264 @ 0x3a218c0] ct_type:0 pic_struct:1
[h264 @ 0x3a218c0] illegal memory management control operation 17
[h264 @ 0x3a218c0] MBAFF frame? aye
[h264 @ 0x3a218c0] Missing reference picture, default is 65975
...

And then you will get reference picture count/missing related errors for the rest of the decode.

[Carl Eugen Hoyos]

Not a regression afaict, the following works fine:

$ ffmpeg -skip_initial_bytes 2000000 -i 2018-08-17-h264_decoder_borks.cut.ts -qscale 2 out.avi

[Carl Eugen Hoyos]

Works fine with xine;-)

[jeeb]

Yes, this is not a regression (at least not a recent one) since I've poked at this with various versions from the last year+ or so.

The problem currently is that this glitch or whatever it is happens relatively often in certain streams (although I was only recently able to capture it as it happened), and the decoder doesn't die hard enough, or re-initialize its state properly - thus leading to artifacts in long-running decoders for the rest of the session. Let's just say that's not optimal. That's the primary reason why I created this ticket.

I tested with VAAPI in mpv, and it bails out at the glitch point and falls back to swdec. Of course the next time the glitch happens, swdec is left in a state that isn't optimal.

[Carl Eugen Hoyos]

Replying to JEEB:

Yes, this is not a regression (at least not a recent one) since I've poked at this with various versions from the last year+ or so.

Yesterday, I tested various versions from the last decade.

[Balling]

Oh, wow!! This causes a bug in nvidia ffplay -vcodec h264_cuvid (this comes from h264 decoder though).

[NULL @ 000002f3282cbe80] illegal memory management control operation 17
[NULL @ 000002f3282cbe80] illegal memory management control operation 17

looks like #5533

[Balling]

illegal memory management control operation 17

is also printed with trace_headers

memory_management_control_operation out of range: 17, but must be in [0,6].

see ​https://webrtc.googlesource.com/src/webrtc/+/f54860e9ef0b68e182a01edc994626d21961bc4b/common_video/h264/h264_bitstream_parser.cc

​https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/d3d12umddi/ns-d3d12umddi-d3d12ddi_video_encoder_picture_control_codec_data_h264_reference_picture_marking_operation_0082_0