avformat/mpegtsenc: mpegts_check_bitstream broken since 8559cce3c37b (wrong operator precedence)

[Sebastian Meyer]

Summary of the bug:
Commit ​8559cce3c37b refactored mpegtsenc's mpegts_check_bitstream function and added a codec-id lookup table instead of checking each codec-id individually. In the process, a uint8_t bitmask was applied to the stream's st->codecpar->extradata[0] which was intended to be checked against the lookup table's value field.

However, the commit author and the reviewers made a mistake and didn't notice the incorrect operator precedence of the bitwise-and and equality comparison.

Beginning from this commit until today's HEAD commit on the master branch, muxing MPEG-TS fails for certain inputs.

This issue was originally noticed here:
​https://github.com/streamlink/streamlink/issues/5876

How to reproduce:
In the issue thread linked above, there's sample data of two MPEG-TS streams, one video and one audio stream. Remuxing into one MPEG-TS stream fails due to this bug. Builds of earlier commits work fine.
​https://github.com/streamlink/streamlink/files/14537316/5876.zip

The error messages are this:

[mpegts @ 0x6392afb61240] h264 bitstream error, startcode missing, size 100902 data 00018A22
[mpegts @ 0x6392afb61240] h264 bitstream error, startcode missing, size 539 data 00000217
[mpegts @ 0x6392afb61240] h264 bitstream error, startcode missing, size 165 data 000000A1
...

Since I'm not familiar with mailing lists and since I didn't find the bug myself, I'm not going to send a patch. The person who found this bug in the linked issue thread apparently wanted to submit a patch, but hasn't responded since, so I decided to open this bug report instead. Maybe some of the responsible devs want to have a look at this again anyway.

Thanks.

[Gyan]

I've sent a patch.

[Gyan]

Fixed in f5441e441f9b0d235e49bdccc69e141ccd92e854