sws_scale crashes after deinterlacing with avpicture_deinterlace (only on recent versions of FFmpeg)

[Andrea3000]

I use FFmpeg API in my Mac OS X application to decode video frames.
In case of interlaced frame I'm used to:
1) convert frame to YUV420P with sws_scale
2) deinterlace frame with avpicture_deinterlace
3) convert frame back to my required pixel format (UYVY422) with sws_scale
(I obviuosly use two different swscale context)

This has always worked fine with git development branch of FFmpeg
till end of october.

On 11th of november I updated FFmpeg to the latest version
available on git repository and since then (till today's version of FFmpeg)
my app crash while performing deinterlace and picture format conversion.

If I convert frame to YUV420P and then back to UYVY422 (without deinterlacing),
it works fine but frame is obviously interlaced.

If I convert frame to YUV420P and then deinterlace it (without back converting to
UYVY422), the app doesn't crash but the output is obviously only a green tint.

It's only the combination of all three steps (conversion, deinterlacing and back
conversion) which makes my app crash. Debugging the crash I discovered that
it is caused by yv12touyvy_MMX2 function but, unfortunately, that function has
not been changed from end of october till now..

It happens with every movie file.

Since I'm using FFmpeg as API, I'm not able to provide further informations..or I don't know how to do it at least..

[Carl Eugen Hoyos]

Please find the commit that introduced the problem and please provide backtrace etc. as explained on ​http://ffmpeg.org/bugreports.html

[Andrea3000]

Replying to cehoyos:

Please find the commit that introduced the problem and please provide backtrace etc. as explained on ​http://ffmpeg.org/bugreports.html

Sorry for the delay but it tooks me hours to figure out which was the incriminated commit but finally I have found it!
This is the commit which introduced the issue: 13b7781ec8d475513c1ee40a6e481763b728a71e

For what is about the backtrace, I don't understand how can I use "gdb ffmpeg_g" as long as I'm using FFmpeg only as an API..
Sorry but I'm a newbie..

[Andrea3000]

Ok, I added gdb output, backtrace and disassembly

[Michael Niedermayer]

You could try valgrind or provide a compileable testcase.

[Andrea3000]

Replying to michael:

You could try valgrind or provide a compileable testcase.

I apologise in advanced if the informations I'm going to report aren't useful. I'm a newbie developer and I've still to learn a lot of stuff..
I paste here the relevant output from valgrind:

==27261== 1 errors in context 1 of 8:
==27261== Thread 1:
==27261== Use of uninitialised value of size 8
==27261==    at 0x1021E803D: ff_h264_decode_seq_parameter_set (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libavcodec.dylib)
==27261==    by 0x7FFF5FBFDCAF: ???
==27261==    by 0x10820C8BF: ???
==27261==    by 0x7FFF5FBFDCB0: ???
==27261==    by 0x38003E9F: ???
==27261==    by 0x11B233E9F: ???
==27261== 
==27261== 
==27261== 1 errors in context 2 of 8:
==27261== Conditional jump or move depends on uninitialised value(s)
==27261==    at 0x1021E802B: ff_h264_decode_seq_parameter_set (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libavcodec.dylib)
==27261==    by 0x7FFF5FBFDCAF: ???
==27261==    by 0x10820C8BF: ???
==27261==    by 0x7FFF5FBFDCB0: ???
==27261==    by 0x38003E9F: ???
==27261==    by 0x11B233E9F: ???
==27261== 
==27261== 
==27261== 1 errors in context 3 of 8:
==27261== Conditional jump or move depends on uninitialised value(s)
==27261==    at 0x1021E7FCA: ff_h264_decode_seq_parameter_set (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libavcodec.dylib)
==27261==    by 0x7FFF5FBFDCAF: ???
==27261==    by 0x10820C8BF: ???
==27261==    by 0x7FFF5FBFDCB0: ???
==27261==    by 0x38003E9F: ???
==27261==    by 0x11B233E9F: ???
==27261== 
==27261== 
==27261== 1 errors in context 4 of 8:
==27261== Conditional jump or move depends on uninitialised value(s)
==27261==    at 0x1021E7F81: ff_h264_decode_seq_parameter_set (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libavcodec.dylib)
==27261==    by 0x7FFF5FBFDCAF: ???
==27261==    by 0x10820C8BF: ???
==27261==    by 0x7FFF5FBFDCB0: ???
==27261==    by 0x38003E9F: ???
==27261==    by 0x11B233E9F: ???
==27261== 
==27261== 
==27261== 3786240 errors in context 5 of 8:
==27261== Thread 9:
==27261== Invalid read of size 8
==27261==    at 0x102C50D84: yv12touyvy_MMX2 (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libswscale.dylib)
==27261==    by 0x43F: ???
==27261==  Address 0x12b9b6088 is 8 bytes inside a block of size 3,110,400 free'd
==27261==    at 0x1001A552D: free (vg_replace_malloc.c:430)
==27261==    by 0x10009A445: -[VideoTrack decodePacket] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x10009AD7F: -[VideoTrack decodeThreadFunc:] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x1001EA113: __NSThread__main__ (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==27261==    by 0x102FFAFD5: _pthread_start (in /usr/lib/libSystem.B.dylib)
==27261==    by 0x102FFAE88: thread_start (in /usr/lib/libSystem.B.dylib)
==27261== 
==27261== 
==27261== 3786240 errors in context 6 of 8:
==27261== Invalid read of size 8
==27261==    at 0x102C50D80: yv12touyvy_MMX2 (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libswscale.dylib)
==27261==    by 0x43F: ???
==27261==  Address 0x12b9b6080 is 0 bytes inside a block of size 3,110,400 free'd
==27261==    at 0x1001A552D: free (vg_replace_malloc.c:430)
==27261==    by 0x10009A445: -[VideoTrack decodePacket] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x10009AD7F: -[VideoTrack decodeThreadFunc:] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x1001EA113: __NSThread__main__ (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==27261==    by 0x102FFAFD5: _pthread_start (in /usr/lib/libSystem.B.dylib)
==27261==    by 0x102FFAE88: thread_start (in /usr/lib/libSystem.B.dylib)
==27261== 
==27261== 
==27261== 3786240 errors in context 7 of 8:
==27261== Invalid read of size 8
==27261==    at 0x102C50D76: yv12touyvy_MMX2 (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libswscale.dylib)
==27261==    by 0x43F: ???
==27261==  Address 0x12bc2ed80 is 2,592,000 bytes inside a block of size 3,110,400 free'd
==27261==    at 0x1001A552D: free (vg_replace_malloc.c:430)
==27261==    by 0x10009A445: -[VideoTrack decodePacket] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x10009AD7F: -[VideoTrack decodeThreadFunc:] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x1001EA113: __NSThread__main__ (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==27261==    by 0x102FFAFD5: _pthread_start (in /usr/lib/libSystem.B.dylib)
==27261==    by 0x102FFAE88: thread_start (in /usr/lib/libSystem.B.dylib)
==27261== 
==27261== 
==27261== 3786240 errors in context 8 of 8:
==27261== Invalid read of size 8
==27261==    at 0x102C50D6F: yv12touyvy_MMX2 (in <path/to/my/app>/MyPlayer.app/Contents/Frameworks/libswscale.dylib)
==27261==    by 0x43F: ???
==27261==  Address 0x12bbb0480 is 2,073,600 bytes inside a block of size 3,110,400 free'd
==27261==    at 0x1001A552D: free (vg_replace_malloc.c:430)
==27261==    by 0x10009A445: -[VideoTrack decodePacket] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x10009AD7F: -[VideoTrack decodeThreadFunc:] (in <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer)
==27261==    by 0x1001EA113: __NSThread__main__ (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==27261==    by 0x102FFAFD5: _pthread_start (in /usr/lib/libSystem.B.dylib)
==27261==    by 0x102FFAE88: thread_start (in /usr/lib/libSystem.B.dylib)

This output is the result of about 1 second of movie playback and it has been generated with:

valgrind -v --error-limit=no --dsymutil=yes --alignment=16 --leak-check=full <path/to/my/app>/MyPlayer.app/Contents/MacOS/MyPlayer

Are these the information you need or I have to run valgrind with different command line options?

[Andrea3000]

I have discovered another "bug" and I'm quite sure that it is related to the bug explained in this ticket and can be the cause of it.

Take a look at this two picture which come from a progressive movie (therefore no deinterlacing and no crash):
1) Prior to incriminated commit: ​Image1
2) After incriminated commit: ​Image2

As you can see, after incriminated commit, height of the frame is 8 pixels bigger than the correct one (in this case 1088 against 1080), and a black border appears in the bottom part of the window/frame.
This happens with every h264 stream inside m2ts container and never happens with mkv container.
MPEG-2 and VC-1 streams don't have this issue.
This happens even with interlaced movie (with deinterlacing disabled).
When deinterlacing is enabled, that black border is yellow/green in the first frame and after that the app crash.

All of the interlaced movie file that I have are h264, but I'm quite sure that MPEG-2 and VC-1 video stream won't lead to any crash when deinterlacing.
I have deinterlaced progressive MPEG-2 and VC-1 (it's useless but possible, I think) and the app doesn't crash at all. On the contrary, if I try to deinterlace a progressive h264, the app crashes.

So I'm quite sure that the issue is related to the bug that add those 8 pixels in the height of h264 frames inside m2ts containers, since only those tracks make the app crashes when deinterlacing.

[Michael Niedermayer]

Valgrind indicates that you run sws on memory that has been freed.
Its output should be a bit more informative if the involved parts are compiled without optimizations and with debug symbols

[Andrea3000]

Replying to michael:

Valgrind indicates that you run sws on memory that has been freed.
Its output should be a bit more informative if the involved parts are compiled without optimizations and with debug symbols

Thank you for your help. I have found the cause of this issue (but not the solution) and I have opened a specific ticket: https://ffmpeg.org/trac/ffmpeg/ticket/731
Have you got any hint on that bug?

[Michael Niedermayer]

Closing ticket as Ticket731 has been closed 3 month ago, and this appears to be due to use of the wrong width/height fields by the user.
If some issue in ffmpeg remains, please reopen!