libswscale does incorrect range conversion for RGB->YUV and RGB->RGB

[gjdfgh]

libswscale seems to do completely confused range conversions for RGB->YUV. RGB->RGB randomly ignores the source range (at least in the unscaled case), and always ignores the destination range. YUV->RGB ignores destination range as well. The only cases that appear to be working as expected are conversions with YUV->YUV.

Here's the output of a test program, that scales a solid image filled with gray=200, using various colorspace and range parameters. "200" is the input value, and the number after that the retrieved luma value or that of a RGB component. "no-scale" means it has been scaled with same source and destination width/height. The test program is attached as swscale.c to this ticket.

no-scale rgb-lim rgb-lim  200 -> 200
no-scale rgb-lim rgb-ful  200 -> 200
no-scale rgb-lim yuv-lim  200 -> 188
no-scale rgb-lim yuv-ful  200 -> 200
no-scale rgb-ful rgb-lim  200 -> 200
no-scale rgb-ful rgb-ful  200 -> 200
no-scale rgb-ful yuv-lim  200 -> 177
no-scale rgb-ful yuv-ful  200 -> 188
no-scale yuv-lim rgb-lim  200 -> 214
no-scale yuv-lim rgb-ful  200 -> 214
no-scale yuv-lim yuv-lim  200 -> 200
no-scale yuv-lim yuv-ful  200 -> 214
no-scale yuv-ful rgb-lim  200 -> 200
no-scale yuv-ful rgb-ful  200 -> 200
no-scale yuv-ful yuv-lim  200 -> 188
no-scale yuv-ful yuv-ful  200 -> 200

scale    rgb-lim rgb-lim  200 -> 200
scale    rgb-lim rgb-ful  200 -> 200
scale    rgb-lim yuv-lim  200 -> 188
scale    rgb-lim yuv-ful  200 -> 200
scale    rgb-ful rgb-lim  200 -> 188
scale    rgb-ful rgb-ful  200 -> 188
scale    rgb-ful yuv-lim  200 -> 177
scale    rgb-ful yuv-ful  200 -> 188
scale    yuv-lim rgb-lim  200 -> 214
scale    yuv-lim rgb-ful  200 -> 214
scale    yuv-lim yuv-lim  200 -> 200
scale    yuv-lim yuv-ful  200 -> 214
scale    yuv-ful rgb-lim  200 -> 200
scale    yuv-ful rgb-ful  200 -> 200
scale    yuv-ful yuv-lim  200 -> 188
scale    yuv-ful yuv-ful  200 -> 200

YUV->RGB ignores destination ranges, and always assumes full destination range.

RGB->YUV looks completely messed up. Maybe someone flipped a bit somewhere?

Scaled RGB->RGB ignores the destination range, and always seems to assume limitted range (so that rgb-lim->* does no conversion). Unscaled RGB->RGB ignores ranges completely, at least when the same pixel format is used.

If there's some logic behind this, I don't get it, and it's not documented.

If libswscale simply doesn't support limitted RGB range (which would lower its usefulness), at least the RGB->RGB case must be fixed.

[gjdfgh]

Test program

[Michael Niedermayer]

There exists just one 8bit RGB range and that is 0..255, the difference in ranges is specific to YUV. I think thats the only source of errors in the table so iam closing this, but if theres another problem that i missed, please reopen.
Also ive changed sws so it ignores the range for RGB instead of doing something quite random if such combination is used

[gjdfgh]

Limited RGB does seem to exist: for RGB output to TVs and projectors.

Having no limited RGB support is probably better to breaking completely unrelated conversions, though.

[Balling]

Can we reopen this for limited RGB support in the future?

[Balling]

Support for limited range rgb is needed. I do not care what you say. If input limited range rgb is working, then why output does not?

[Balling]

We need to start by either removing the deprecation of yuvj formats or by removing them alltogether (there is patchset from Paul, see ​https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=&submitter=&state=&q=Yuvj&archive=&delegate=). I vote for the first one. It is a joke that even opening JPEG (since jpeg does not support new method with setting range not by pixel_format, also see a funny bug in mpv, ​https://github.com/mpv-player/mpv/issues/9053#issuecomment-1046052804) causes a warning for every tile or whatever! It is nuts! Warnings obviously make it slower out of nothing.

[chris319]

See the following documents:

​​https://tech.ebu.ch/docs/r/r103.pdf

​​https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf

[Carl Eugen Hoyos]

Please do not reopen ancient tickets.

[Balling]

Please do not reopen ancient tickets.

​https://w3c.github.io/PNG-spec/#cICP-chunk

PNG cICP now requires limited range RGB. HAHAHA, wasted.

[Elon Musk]

It supports both, 0 is limited, 1 is full.

[Balling]

It supports both, 0 is limited, 1 is full.

Indeed. And we do not.

[Balling]

Yep. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230131231254.277086-1-leo.izen@gmail.com/