h264 video frames inside m2ts container get altered after commit 13b7781ec8d475513c1ee40a6e481763b728a71e

[Andrea3000]

After commit 13b7781ec8d475513c1ee40a6e481763b728a71e, h264 video frames inside m2ts container are altered.
Take a look at those 2 pictures:
Pre commit: ​Image pre-commit
After commit: ​Image after-commit

As you can see, after that commit, height of the frame is bigger (by 8 pixles) than the correct one (in this case 1088 against 1080), therefore a black border appears in the bottom part of the window/frame.
This happens with every h264 stream inside m2ts container and never happens with mkv container.
MPEG-2 and VC-1 streams don't have this issue both in m2ts and mkv container.

This bug also causes a crash when deinterlacing and converting image with avpicture_deinterlace and sws_scale. (See https://ffmpeg.org/trac/ffmpeg/ticket/713)

[Carl Eugen Hoyos]

Please provide a sample that allows to reproduce the problem (images are nearly always unneeded).
If the sample is larger than 2.5MB, please see ​http://ffmpeg.org/bugreports.html or upload to ​http://www.datafilehost.com/ (and provide the download link).

For all problem reports on this tracker, please provide a (failing) command line and complete, uncut console output.

[Andrea3000]

Replying to cehoyos:

Please provide a sample that allows to reproduce the problem (images are nearly always unneeded).
If the sample is larger than 2.5MB, please see ​http://ffmpeg.org/bugreports.html or upload to ​http://www.datafilehost.com/ (and provide the download link).

For all problem reports on this tracker, please provide a (failing) command line and complete, uncut console output.

It happens with every h264 movie file inside m2ts container, anyway I have uploaded a short sample here: ​http://www.datafilehost.com/download-902f38c6.html

I'm not able to provide the failing command line because I use FFmepg library as API inside my OSX app.

The strange thing is that it seems to be a parsing issue rather then a decoding issue. I have tried using FFmpeg first only as a parser (decoding frames with the GPU) and later even as a decoder and this black border appears in both of the case. Prior to that commit, both software (FFmpeg) and hardware decoding wasn't giving any black border.

[Andrea3000]

I have made a further discovery.
Incorrect frame height is given by AVCodecContext->coded_height which reports a value of 1088.
If I override this value and set it to 1080, the black border is gone and deinterlacing works again.
So now, why does AVCodecContext->coded_height reports a wrong value after that commit?

[Michael Niedermayer]

The width and height of the outputed picture is stored in width/height.
H.264 allows croping an image and the uncropped dimensions are in coded_width/coded_height.
This behavior exists since 29a29226. The quoted commit 13b7781ec8d4 is a merge commit and thus has multiple
ancestors, thus iam not sure if you meant to say both its ancestors work and it is the first not working or if you tested something else

[Andrea3000]

Replying to michael:

The width and height of the outputed picture is stored in width/height.
H.264 allows croping an image and the uncropped dimensions are in coded_width/coded_height.
This behavior exists since 29a29226. The quoted commit 13b7781ec8d4 is a merge commit and thus has multiple
ancestors, thus iam not sure if you meant to say both its ancestors work and it is the first not working or if you tested something else

I've tested f2bd8a0786 ancestor and codec_height was reported correctly. To be precise, I've checked out git repository till b8f02f5b4e which is the last commit prior to 13b7781ec8d4 and that last commit has f2bd8a0786 as ancestor.
Have I to try the second ancestor?

[Michael Niedermayer]

The 2 ancestors of 13b7781ec8d4 are ada8d48 and f2bd8a0
git bisect should find the commit introducing the problem but please first verify that you use width/height and not coded_width/coded_height

[Andrea3000]

Replying to michael:

The 2 ancestors of 13b7781ec8d4 are ada8d48 and f2bd8a0
git bisect should find the commit introducing the problem but please first verify that you use width/height and not coded_width/coded_height

I'm using both width/height and coded_width/coded_height to keep track of both of them.
When I call sws_scale, for example, I pass coded_height as srcSliceH and it has been always worked good. Is it wrong?

The problem is only on coded_width/coded_height, width/height are still correct.

[Andrea3000]

Can someone answer my last question so that I can further investigate which exact commit introduced the problem?

[reimar]

I am no longer 100% sure about what the meaning of each is exactly, but you should not (and should not have a reason to) use the coded_* variants. I can't see how your code could even work with "lowres" if you are using coded_*.

[Carl Eugen Hoyos]

Could you confirm if your issue is fixed if you use width and height instead of coded_width and coded_height?

[Carl Eugen Hoyos]

Please reopen if there still is any problem that you can reproduce.