color_range flag for h264 decode ignored

[onejaguar]

Summary of the bug:
Using "-color_range 2" on an h264 compressed input file should force the full range ("JPEG range") of the values to be decoded, but only the legal range ("MPEG range") actually gets decoded, because the -color_range flag is ignored and overridden by metadata in the file header (even if these is no relevant metadata!). The command-line flag should supersede the metadata.

The attached h264_jpeg_color_range.mov has values from 0-255.

$ ffmpeg -color_range 2 -i h264_jpeg_color_range.mov -f image2 test%05d.jpg

Produces a JPEG which clips the input values at 16-235 and scales them to 0-1.

If I comment out this line in h264.c:
h->avctx->color_range = h->sps.full_range>0 ? AVCOL_RANGE_JPEG : AVCOL_RANGE_MPEG;

Recompile and run the above command I get a JPEG which has the full 0-1 range as expected.

Console output:
./ffmpeg -color_range 2 -i h264_jpeg_color_range.mov -f image2 test%05d.jpg
ffmpeg version N-59945-g26800e3 Copyright (c) 2000-2014 the FFmpeg developers

built on Jan 18 2014 09:50:54 with Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
configuration: --prefix=/usr/local --enable-gpl --enable-nonfree --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid
libavutil 52. 62.100 / 52. 62.100
libavcodec 55. 48.101 / 55. 48.101
libavformat 55. 24.100 / 55. 24.100
libavdevice 55. 5.102 / 55. 5.102
libavfilter 4. 1.100 / 4. 1.100
libswscale 2. 5.101 / 2. 5.101
libswresample 0. 17.104 / 0. 17.104
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/peterp/Desktop/1280x720/h264_jpeg_color_range.mov':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2014-01-18 20:12:51

Duration: 00:00:01.00, start: 0.000000, bitrate: 203 kb/s

Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720, 186 kb/s, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 24k tbn, 48k tbc (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

[swscaler @ 0x7fbc9a00a600] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/Users/peterp/Desktop/1280x720/test%05d.jpg':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
encoder : Lavf55.24.100
Stream #0:0(eng): Video: mjpeg, yuvj420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 23.98 tbc (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)

Press [q] to stop, ? for help
frame= 24 fps=0.0 q=0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A
video:803kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.002674%

Console output with line in h264.c commented out:
./ffmpeg -color_range 2 -i h264_jpeg_color_range.mov -f image2 test%05d.jpg
ffmpeg version N-59945-g26800e3 Copyright (c) 2000-2014 the FFmpeg developers

built on Jan 18 2014 09:50:54 with Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
configuration: --prefix=/usr/local --enable-gpl --enable-nonfree --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid
libavutil 52. 62.100 / 52. 62.100
libavcodec 55. 48.101 / 55. 48.101
libavformat 55. 24.100 / 55. 24.100
libavdevice 55. 5.102 / 55. 5.102
libavfilter 4. 1.100 / 4. 1.100
libswscale 2. 5.101 / 2. 5.101
libswresample 0. 17.104 / 0. 17.104
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/peterp/Desktop/1280x720/h264_jpeg_color_range.mov':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2014-01-18 20:12:51

Duration: 00:00:01.00, start: 0.000000, bitrate: 203 kb/s

Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1280x720, 186 kb/s, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 24k tbn, 48k tbc (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

Output #0, image2, to '/Users/peterp/Desktop/1280x720/test%05d.jpg':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
encoder : Lavf55.24.100
Stream #0:0(eng): Video: mjpeg, yuvj420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 23.98 tbc (default)
Metadata:

creation_time : 2014-01-18 20:12:51
handler_name : Apple Alias Data Handler
timecode : 80:20:08:00

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)

Press [q] to stop, ? for help
frame= 24 fps=0.0 q=0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A
video:815kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.002636%

[onejaguar]

h246 compressed grey ramps with 0-255 code values

[Kieran Kunhya]

The colour range flag is often wrong.

[Carl Eugen Hoyos]

For the attached sample, the option -color_range has never worked, for other samples - like fate-suite/h264/extreme-plane-pred.h264 - the option had an effect until 5dc6bd86

[Michael Niedermayer]

you can override the range using "-vf scale=in_range=jpeg" (needs latest ffmpeg)

the input color_range parameter currently only sets the range when the input file does not specify anything
iam not sure if it would really be more usefull if it would hard override as theres already a hard override available through the scale filter
Thus imo theres no bug but if people disgaree and want the input_range changed, dont hesitate to reopen