swscale produces bright green when going from grayf32 to gbrpf32

[Leo Izen]

Summary of the bug:
Swscale produces bright green RGB data when converting from grayf32 to gbrpf32.

How to reproduce:

$ ffmpeg -i rose.pfm -vf scale,format=gbrpf32le rose2.pfm
ffmpeg version N-92323-g5230257ea1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.1 (GCC) 20230801
  configuration: --prefix=/home/leo/.local --enable-gpl --enable-version3 --enable-nonfree --enable-shared --disable-static --disable-htmlpages --enable-manpages --disable-podpages --disable-txtpages --enable-frei0r --enable-gcrypt --enable-gmp --enable-gnutls --enable-lcms2 --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcelt --enable-libcdio --enable-libdav1d --enable-libdc1394 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libkvazaar --enable-libmodplug --enable-libmp3lame --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librtmp --enable-librubberband --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libzimg --enable-libzmq --enable-openal --enable-opengl --disable-openssl --enable-pic --enable-ffnvcodec --enable-libdrm --enable-vapoursynth --enable-vulkan --extra-cflags='-Wno-format-truncation -Wno-stringop-overflow -Wno-array-bounds'
  libavutil      58. 32.100 / 58. 32.100
  libavcodec     60. 35.100 / 60. 35.100
  libavformat    60. 18.100 / 60. 18.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 14.100 /  9. 14.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
Input #0, pfm_pipe, from 'rose.pfm':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: pfm, grayf32le, 70x46, 25 fps, 25 tbr, 25 tbn
File 'rose2.pfm' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (pfm (native) -> pfm (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'rose2.pfm':
  Metadata:
    encoder         : Lavf60.18.100
  Stream #0:0: Video: pfm, gbrpf32le(pc, gbr/unknown/unknown, progressive), 70x46, q=2-31, 200 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.35.100 pfm
[image2 @ 0x55f568894100] The specified filename 'rose2.pfm' does not contain an image sequence pattern or a pattern is invalid.
[image2 @ 0x55f568894100] Use a pattern such as %03d for an image sequence or use the -update option (with -frames:v 1 if needed) to write a single image.
[out#0/image2 @ 0x55f568893340] video:38kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x    

Samples attached.

[Balling]

ffplay -f rawvideo -video_size 70x46 -pixel_format gbrpf32le -i rose3.rgb

same hapopens

ffmpeg -i C:/Users/ZAQU/Downloads/rose.pfm -vf scale kafca1.rgb

prints even stranger bug:

[rawvideo @ 0000018d2f3d1d40] Packet corrupt (stream = 0, dts = 0).
[rawvideo @ 0000018d2f3d1d40] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from 'kafca1.rgb':
  Duration: N/A, start: 0.000000, bitrate: 7728 kb/s
  Stream #0:0: Video: rawvideo, gbrpf32le, 70x46, 7728 kb/s, 25 tbr, 25 tbn
[rawvideo @ 0000018d2f59fbc0] Invalid buffer size, packet size 12880 < expected frame_size 38640

zscale does not help

[Niklas Haas]

I had a look at this, and I think the error is down to a confusion about what internal formats swscale is using.

In input.c and output.c and many other places, swscale follows the rule of using 15-bit intermediate if output bpc is <= 8, and 19-bit (inside int32_t) intermediate otherwise. See e.g. the comments on hyScale() on swscale_internal.h. These are also the coefficients that yuv2gbrpf32_full_X_c() is using.

In contrast to this, the plane init code in slice.c (function fill_ones) is assuming that we use 35-bit intermediates (inside 64-bit integers) for this case, seemingly added by commit b4967fc71c63eae8cd96f9c46cd3e1fbd705bbf9 with no further justification.

As such, I propose simply dropping this case from fill_ones(), as 64-bits precision is way more than conceivably necessary for any realistic use case, and even 32 bits of precision is arguably overkill for anything other than maintaining bit exactness with some abstract mathematical reference. (Something that would require also extending the precision of all conversion tables, yuv coefficients, scaling filters and so on)

[Niklas Haas]

Fixed in commit ce457bfccdd2558dc0e508f2ddf9925e1b741209