Aspect ratio change in the middle of a video stream

[Arthur Marxer]

I have a .ts file with an aspect ratio change in the middle of the video stream. When I play the file with e.g. VLC, then the VLC player window changes its size in the middle of the stream as expected.
Now, I want to extract image snapshots (jpeg images) from the video with a height of 200 pixel and the according widht. I expect that the widhts of the produced images will change according to the video source aspect ratio. But all images produced by ffmpeg have the same size. The problem persists if I remove the scale filter.

How to reproduce:

ffmpeg.exe -i aspect_ratio_change.ts -vf fps=1,scale=-1:200 out_%3d.jpeg
ffmpeg version N-63013-g4cdea92 Copyright (c) 2000-2014 the FFmpeg developers
  built on May  6 2014 22:02:02 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      52. 81.100 / 52. 81.100
  libavcodec     55. 60.103 / 55. 60.103
  libavformat    55. 37.102 / 55. 37.102
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mpegts, from 'aspect_ratio_change.ts':
  Duration: 00:00:07.43, start: 1978.640000, bitrate: 4715 kb/s
  Program 258
    Stream #0:0[0xe0]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 720x576 [SAR 16:15 DAR 4:3], max. 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0xc0](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 186 kb/s (clean effects)
[swscaler @ 02d57000] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'out_%3d.jpeg':
  Metadata:
    encoder         : Lavf55.37.102
    Stream #0:0: Video: mjpeg, yuvj420p, 250x200 [SAR 16:15 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 1 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video -> mjpeg)
Press [q] to stop, [?] for help
frame=    8 fps=0.0 q=1.6 Lsize=N/A time=00:00:08.00 bitrate=N/A
video:106kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17044493962313728.000000%

Here is the output with loglevel 99:

D:\media\ffmpegtest>C:\Programme\ffmpeg\ffmpeg.exe -v 9 -loglevel 99 -i aspect_ratio_change.ts -vf fps=1,scale=-1:200 out_%3d.jpeg
ffmpeg version N-63013-g4cdea92 Copyright (c) 2000-2014 the FFmpeg developers
  built on May  6 2014 22:02:02 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libblu
ray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      52. 81.100 / 52. 81.100
  libavcodec     55. 60.103 / 55. 60.103
  libavformat    55. 37.102 / 55. 37.102
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'aspect_ratio_change.ts'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'fps=1,scale=-1:200'.
Reading option 'out_%3d.jpeg' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input file aspect_ratio_change.ts.
Successfully parsed a group of options.
Opening an input file: aspect_ratio_change.ts.
[mpegts @ 02d54380] Format mpegts probed with size=2048 and score=100
[mpegts @ 02d54380] stream=0 stream_type=2 pid=e0 prog_reg_desc=
[mpegts @ 02d54380] stream=1 stream_type=3 pid=c0 prog_reg_desc=
[mpegts @ 02d54380] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0
[mpegts @ 02d54380] All programs have pmt, headers found
[mpegts @ 02d54380] All info found
[mpegts @ 02d54380] After avformat_find_stream_info() pos: 0 bytes read:675984 seeks:2 frames:56
Input #0, mpegts, from 'aspect_ratio_change.ts':
  Duration: 00:00:07.43, start: 1978.640000, bitrate: 4715 kb/s
  Program 258
    Stream #0:0[0xe0], 21, 1/90000: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 720x576 [SAR 16:15 DAR 4:3], 1/50, max. 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0xc0](deu), 35, 1/90000: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 186 kb/s (clean effects)
Successfully opened the file.
Parsing a group of options: output file out_%3d.jpeg.
Applying option vf (set video filters) with argument fps=1,scale=-1:200.
Successfully parsed a group of options.
Opening an output file: out_%3d.jpeg.
Successfully opened the file.
detected 2 logical cores
[Parsed_fps_0 @ 003f8fa0] Setting 'fps' to value '1'
[Parsed_fps_0 @ 003f8fa0] fps=1/1
[Parsed_scale_1 @ 003fcf20] Setting 'w' to value '-1'
[Parsed_scale_1 @ 003fcf20] Setting 'h' to value '200'
[Parsed_scale_1 @ 003fcf20] Setting 'flags' to value '0x4'
[Parsed_scale_1 @ 003fcf20] w:-1 h:200 flags:'0x4' interl:0
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'video_size' to value '720x576'
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'time_base' to value '1/90000'
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'pixel_aspect' to value '16/15'
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 02d541e0] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 02d541e0] w:720 h:576 pixfmt:yuv420p tb:1/90000 fr:25/1 sar:16/15 sws_param:flags=2
[format @ 02d682e0] compat: called with args=[yuvj420p|yuvj422p|yuvj444p]
[format @ 02d682e0] Setting 'pix_fmts' to value 'yuvj420p|yuvj422p|yuvj444p'
[AVFilterGraph @ 003fb060] query_formats: 5 queried, 4 merged, 0 already done, 0 delayed
[Parsed_scale_1 @ 003fcf20] picking yuvj420p out of 3 ref:yuv420p alpha:0
[swscaler @ 02d57020] deprecated pixel format used, make sure you did set range correctly
[Parsed_scale_1 @ 003fcf20] w:720 h:576 fmt:yuv420p sar:16/15 -> w:250 h:200 fmt:yuvj420p sar:16/15 flags:0x4
[mjpeg @ 02dcc580] Forcing thread count to 1 for MJPEG encoding, use -thread_type slice or a constant quantizer if you want to use multiple cpu cores
[mjpeg @ 02dcc580] intra_quant_bias = 96 inter_quant_bias = 0
Output #0, image2, to 'out_%3d.jpeg':
  Metadata:
    encoder         : Lavf55.37.102
    Stream #0:0, 0, 1/90000: Video: mjpeg, yuvj420p, 250x200 [SAR 16:15 DAR 4:3], 1/1, q=2-31, 200 kb/s, 90k tbn, 1 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video -> mjpeg)
Press [q] to stop, [?] for help
[mpegts @ 02d54380] Correcting start time by 229956
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 11 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 23 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 23 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 22 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 21 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 23 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 23 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).
    Last message repeated 6 times
[Parsed_fps_0 @ 003f8fa0] Dropping 1 frame(s).07.00 bitrate=N/A
    Last message repeated 8 times
[AVIOContext @ 02d56f00] Statistics: 0 seeks, 1 writeouts
[output stream 0:0 @ 02e0ffa0] EOF on sink link output stream 0:0:default.
No more output streams to write to, finishing.
frame=    8 fps=0.0 q=1.6 Lsize=N/A time=00:00:08.00 bitrate=N/A
video:106kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17044493962313728.000000%
Input file #0 (aspect_ratio_change.ts):
  Input stream #0:0 (video): 182 packets read (3971820 bytes); 177 frames decoded;
  Input stream #0:1 (audio): 0 packets read (0 bytes);
  Total: 182 packets (3971820 bytes) demuxed
Output file #0 (out_%3d.jpeg):
  Output stream #0:0 (video): 8 frames encoded; 8 packets muxed (108227 bytes);
  Total: 8 packets (108227 bytes) muxed
183 frames successfully decoded, 0 decoding errors
[Parsed_fps_0 @ 003f8fa0] 177 frames in, 8 frames out; 169 frames dropped, 0 frames duplicated.
[AVIOContext @ 003fada0] Statistics: 5055632 bytes read, 2 seeks

D:\media\ffmpegtest>

[Arthur Marxer]

First produced image

[Arthur Marxer]

produced image after ratio change

[Arthur Marxer]

I uploaded the sample file to the FTP server

[Carl Eugen Hoyos]

Could you add ffplay console output for the two jpg files you uploaded? And explain (again) what is wrong with them?

[Arthur Marxer]

Yes, of course. Here is the ffplay output of out_001.jpeg:

D:\media\ffmpegtest>C:\Programme\ffmpeg\ffplay.exe -i out_001.jpeg
ffplay version N-63013-g4cdea92 Copyright (c) 2003-2014 the FFmpeg developers
  built on May  6 2014 22:02:02 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable
-decklink --enable-zlib
  libavutil      52. 81.100 / 52. 81.100
  libavcodec     55. 60.103 / 55. 60.103
  libavformat    55. 37.102 / 55. 37.102
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from 'out_001.jpeg': vq=    0KB sq=    0B f=0/0
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc), 250x200 [SAR 16:15 DAR 4:3], 25 tbr
, 25 tbn, 25 tbc
[swscaler @ 02d7d020] deprecated pixel format used, make sure you did set range
correctly
  13.77 M-V:  0.000 fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0

And this is out_007.jpeg

D:\media\ffmpegtest>C:\Programme\ffmpeg\ffplay.exe -i out_007.jpeg
ffplay version N-63013-g4cdea92 Copyright (c) 2003-2014 the FFmpeg developers
  built on May  6 2014 22:02:02 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable
-decklink --enable-zlib
  libavutil      52. 81.100 / 52. 81.100
  libavcodec     55. 60.103 / 55. 60.103
  libavformat    55. 37.102 / 55. 37.102
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from 'out_007.jpeg': vq=    0KB sq=    0B f=0/0
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc), 250x200 [SAR 64:45 DAR 16:9], 25 tb
r, 25 tbn, 25 tbc
[swscaler @ 02d79460] deprecated pixel format used, make sure you did set range
correctly
   2.48 M-V:  0.000 fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0

The aspect ratio in the video changes from 5:4 to 16:9 (a matter of fact).
The first image (out_001.jpeg) is correct. It has a size of 250x200 pixel with an aspect ratio of 5:4.
The second image (out_007.jpeg) is not correct. It is taken after the aspect ratio change and has a wrong size of 250x200. It should have an aspect ratio of 16:9 and a size of 356x200.

[Arthur Marxer]

"Handmade" expected output

[Carl Eugen Hoyos]

Replying to mar:

The aspect ratio in the video changes from 5:4 to 16:9 (a matter of fact).

I believe (as a matter of fact) the aspect changes from 4:3 to 16:9. The same is true for the two jpg files you originally attached - the first has a display aspect ratio of 4:3, the second 16:9: This is what the console output shows and what ffplay should have shown you or am I wrong?

[Arthur Marxer]

Yes, you are right. I have not used ffplay to view jpeg before, and it displays the image correctly.
However, if I view the jpeg in another viewer (e.g. Firefox or Photoshop or the Windows default image viewer) it does not display correctly. And also the pixel size is not correct. It seems that the information of the aspect ratio is preserved in the jpeg file and that ffplay "stretches" the image when it is displayed (stretching also causes a bit of quality loss). But the image is not "stretched" by other viewers. Is this saving of the aspect ratio in the jpeg a "proprietary" feature used by ffmpeg or are all those other vierwers "buggy"?

[Carl Eugen Hoyos]

Replying to mar:

However, if I view the jpeg in another viewer (e.g. Firefox or Photoshop or the Windows default image viewer) it does not display correctly. And also the pixel size is not correct.

So maybe you used the wrong bug tracker?

It seems that the information of the aspect ratio is preserved in the jpeg file and that ffplay "stretches" the image when it is displayed

Note that this is exactly what is done for your input video.

(stretching also causes a bit of quality loss).

The opposite is true afaict (the scaling filter necessary to produce output files with sar==1 is not lossless).

But the image is not "stretched" by other viewers. Is this saving of the aspect ratio in the jpeg a "proprietary" feature used by ffmpeg or are all those other vierwers "buggy"?

It is definitely not "proprietary" but specified for jpeg.
Please use -vf scale=h=ih:w=sar*iw to produce output files with sar==1.
(Sorry if this is not the 100% exact filter command, in any case the scale filter is able to produce output files as you want them.)

[Arthur Marxer]

Thanks for this solution and your precious time!!

[Arthur Marxer]

I made a few tests and have good and bad news: In order to get images with a sar==1, I executed the following command:

ffmpeg -i aspect_ratio_change.ts -vf fps=1,scale=h=ih:w=sar*iw out_%3d.jpeg

The first image (before the aspect ratio change) has 768x576 [SAR 1:1 DAR 4:3] which is perfect!
But the second image (after the change) has 768x576 [SAR 4:3 DAR 16:9] which is not what I expected, it should be 1024x576 [SAR 1:1 DAR 16:9].
It seems that the pixel size of the resulting image is calculated only once at the beginning of the process and it gets not adjusted when an aspect ratio change occurs.

[Arthur Marxer]

Reopen, because the filter scale=h=ih:w=sar*iw does not produce images with sar==1 after an aspect ratio change occured in the middle of a video stream.