-vf maskedmerge results in expected value (-1) for mask values 129 to 255 in 8bit YUV or RGB

[pdr0]

Summary of the bug:
How to reproduce:

"D:\\_DOWNLOADS\\ffmpeg-master-latest-win64-gpl_20220302\\bin\\ffmpeg" -report -i black.png -i white.png -i greyscaleramp.png -filter_complex "[0:0][1:0][2:0]maskedmerge" ffmpegmaskedmerge_20220302.png -y
ffmpeg version N-105822-g4b72bca6ca-20220302 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (crosstool-NG 1.24.0.533_681aaef)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --en  libavutil      57. 22.100 / 57. 22.100
  libavcodec     59. 21.103 / 59. 21.103
  libavformat    59. 17.102 / 59. 17.102
  libavdevice    59.  5.100 / 59.  5.100
  libavfilter     8. 27.100 /  8. 27.100
  libswscale      6.  5.100 /  6.  5.100
  libswresample   4.  4.100 /  4.  4.100
  libpostproc    56.  4.100 / 56.  4.100
Splitting the commandline.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Reading option '-i' ... matched as input url with argument 'black.png'.
Reading option '-i' ... matched as input url with argument 'white.png'.
Reading option '-i' ... matched as input url with argument 'greyscaleramp.png'.
Reading option '-filter_complex' ... matched as option 'filter_complex' (create a complex filtergraph) with argument '[0:0][1:0][2:0]maskedmerge'.
Reading option 'ffmpegmaskedmerge_20220302.png' ... matched as output url.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option report (generate a report) with argument 1.
Applying option filter_complex (create a complex filtergraph) with argument [0:0][1:0][2:0]maskedmerge.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url black.png.
Successfully parsed a group of options.
Opening an input file: black.png.
[NULL @ 0000005d59913bc0] Opening 'black.png' for reading
[file @ 0000005d59a1a140] Setting default whitelist 'file,crypto,data'
[png_pipe @ 0000005d59913bc0] Format png_pipe probed with size=2048 and score=99
[png_pipe @ 0000005d59913bc0] Before avformat_find_stream_info() pos: 0 bytes read:285 seeks:0 nb_streams:1
[png_pipe @ 0000005d59913bc0] After avformat_find_stream_info() pos: 285 bytes read:285 seeks:0 frames:1
Input #0, png_pipe, from 'black.png':
  Duration: N/A, bitrate: N/A
  Stream #0:0, 1, 1/25: Video: png, rgb24(pc), 256x256, 25 fps, 25 tbr, 25 tbn
Successfully opened the file.
Parsing a group of options: input url white.png.
Successfully parsed a group of options.
Opening an input file: white.png.
[NULL @ 0000005d59a1c700] Opening 'white.png' for reading
[file @ 0000005d59a1b380] Setting default whitelist 'file,crypto,data'
[png_pipe @ 0000005d59a1c700] Format png_pipe probed with size=2048 and score=99
[png_pipe @ 0000005d59a1c700] Before avformat_find_stream_info() pos: 0 bytes read:773 seeks:0 nb_streams:1
[png_pipe @ 0000005d59a1c700] After avformat_find_stream_info() pos: 773 bytes read:773 seeks:0 frames:1
Input #1, png_pipe, from 'white.png':
  Duration: N/A, bitrate: N/A
  Stream #1:0, 1, 1/25: Video: png, rgb24(pc), 256x256, 25 fps, 25 tbr, 25 tbn
Successfully opened the file.
Parsing a group of options: input url greyscaleramp.png.
Successfully parsed a group of options.
Opening an input file: greyscaleramp.png.
[NULL @ 0000005d59914800] Opening 'greyscaleramp.png' for reading
[file @ 0000005d59914ac0] Setting default whitelist 'file,crypto,data'
[png_pipe @ 0000005d59914800] Format png_pipe probed with size=2048 and score=99
[png_pipe @ 0000005d59914800] Before avformat_find_stream_info() pos: 0 bytes read:790 seeks:0 nb_streams:1
[png_pipe @ 0000005d59914800] After avformat_find_stream_info() pos: 790 bytes read:790 seeks:0 frames:1
Input #2, png_pipe, from 'greyscaleramp.png':
  Duration: N/A, bitrate: N/A
  Stream #2:0, 1, 1/25: Video: png, rgb24(pc), 256x256 [SAR 1:1 DAR 1:1], 25 fps, 25 tbr, 25 tbn
Successfully opened the file.
Parsing a group of options: output url ffmpegmaskedmerge_20220302.png.
Successfully parsed a group of options.
Opening an output file: ffmpegmaskedmerge_20220302.png.
Successfully opened the file.
detected 8 logical cores
Stream mapping:
  Stream #0:0 (png) -> maskedmerge
  Stream #1:0 (png) -> maskedmerge
  Stream #2:0 (png) -> maskedmerge
  maskedmerge:default -> Stream #0:0 (png)
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[graph 0 input from stream 0:0 @ 0000005d5995b740] Setting 'video_size' to value '256x256'
[graph 0 input from stream 0:0 @ 0000005d5995b740] Setting 'pix_fmt' to value '2'
[graph 0 input from stream 0:0 @ 0000005d5995b740] Setting 'time_base' to value '1/25'
[graph 0 input from stream 0:0 @ 0000005d5995b740] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 0000005d5995b740] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0000005d5995b740] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1
[graph 0 input from stream 1:0 @ 0000005d5995c840] Setting 'video_size' to value '256x256'
[graph 0 input from stream 1:0 @ 0000005d5995c840] Setting 'pix_fmt' to value '2'
[graph 0 input from stream 1:0 @ 0000005d5995c840] Setting 'time_base' to value '1/25'
[graph 0 input from stream 1:0 @ 0000005d5995c840] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 1:0 @ 0000005d5995c840] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 1:0 @ 0000005d5995c840] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1
[graph 0 input from stream 2:0 @ 0000005d5995b240] Setting 'video_size' to value '256x256'
[graph 0 input from stream 2:0 @ 0000005d5995b240] Setting 'pix_fmt' to value '2'
[graph 0 input from stream 2:0 @ 0000005d5995b240] Setting 'time_base' to value '1/25'
[graph 0 input from stream 2:0 @ 0000005d5995b240] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 2:0 @ 0000005d5995b240] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 2:0 @ 0000005d5995b240] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:1/1
[format @ 0000005d5995ca40] Setting 'pix_fmts' to value 'rgb24|rgba|rgb48be|rgba64be|pal8|gray|ya8|gray16be|ya16be|monob'
[auto_scale_0 @ 0000005d5995cc40] w:iw h:ih flags:'' interl:0
[Parsed_maskedmerge_0 @ 0000005d5995c740] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_maskedmerge_0'
[auto_scale_1 @ 0000005d5995cb40] w:iw h:ih flags:'' interl:0
[Parsed_maskedmerge_0 @ 0000005d5995c740] auto-inserting filter 'auto_scale_1' between the filter 'graph 0 input from stream 1:0' and the filter 'Parsed_maskedmerge_0'
[auto_scale_2 @ 0000005d5995b640] w:iw h:ih flags:'' interl:0
[Parsed_maskedmerge_0 @ 0000005d5995c740] auto-inserting filter 'auto_scale_2' between the filter 'graph 0 input from stream 2:0' and the filter 'Parsed_maskedmerge_0'
[auto_scale_3 @ 0000005d5995cd40] w:iw h:ih flags:'' interl:0
[format @ 0000005d5995ca40] auto-inserting filter 'auto_scale_3' between the filter 'Parsed_maskedmerge_0' and the filter 'format'
[AVFilterGraph @ 0000005d5997d9c0] query_formats: 6 queried, 5 merged, 4 already done, 0 delayed
[auto_scale_0 @ 0000005d5995cc40] picking gbrp out of 61 ref:rgb24 alpha:0
[auto_scale_3 @ 0000005d5995cd40] picking rgb24 out of 10 ref:gbrp alpha:0
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5ef8efc0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5ef9bf80] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5efa8f80] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5efb5f40] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5efcaf00] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5efda180] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5efe8440] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5eff6740] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5ef82000] [swscaler @ 0000005d5f01e800] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scale_0 @ 0000005d5995cc40] w:256 h:256 fmt:rgb24 sar:0/1 -> w:256 h:256 fmt:gbrp sar:0/1 flags:0x0
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f03e7c0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f04b780] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f0614c0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f06f1c0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f07cec0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f096c00] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f0a4900] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f0b5e00] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f02d800] [swscaler @ 0000005d5f0c32c0] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scale_1 @ 0000005d5995cb40] w:256 h:256 fmt:rgb24 sar:0/1 -> w:256 h:256 fmt:gbrp sar:0/1 flags:0x0
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d5f0eb280] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d5f108240] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d5f115240] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d5f12a200] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d5f1371c0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d631500c0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d6315d080] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d63172040] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d5f0dc280] [swscaler @ 0000005d6317f040] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scale_2 @ 0000005d5995b640] w:256 h:256 fmt:rgb24 sar:1/1 -> w:256 h:256 fmt:gbrp sar:1/1 flags:0x0
[Parsed_maskedmerge_0 @ 0000005d5995c740] [framesync @ 0000005d5995af88] Selected 1/25 time base
[Parsed_maskedmerge_0 @ 0000005d5995c740] [framesync @ 0000005d5995af88] Sync level 1
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d6319afc0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d631affc0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d631ccf80] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d631e1f80] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d631fef40] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d6320bf40] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d63218f00] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d63225ec0] Forcing full internal H chroma due to input having non subsampled chroma
[swscaler @ 0000005d6318c000] [swscaler @ 0000005d63232e80] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scale_3 @ 0000005d5995cd40] w:256 h:256 fmt:gbrp sar:0/1 -> w:256 h:256 fmt:rgb24 sar:0/1 flags:0x0
Output #0, image2, to 'ffmpegmaskedmerge_20220302.png':
  Metadata:
    encoder         : Lavf59.17.102
  Stream #0:0, 0, 1/25: Video: png, rgb24(pc, gbr/unknown/unknown, progressive), 256x256, q=2-31, 200 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc59.21.103 png
Clipping frame in rate conversion by 0.000008
frame=    1 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[Parsed_maskedmerge_0 @ 0000005d5995c740] [framesync @ 0000005d5995af88] Sync level 0
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[out_0_0 @ 0000005d5995c940] EOF on sink link out_0_0:default.
No more output streams to write to, finishing.
[image2 @ 0000005d59919c00] Opening 'ffmpegmaskedmerge_20220302.png' for writing
[file @ 0000005d599467c0] Setting default whitelist 'file,crypto,data'
[AVIOContext @ 0000005d6542b9c0] Statistics: 1934 bytes written, 0 seeks, 1 writeouts
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.215x    
video:2kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (black.png):
  Input stream #0:0 (video): 1 packets read (285 bytes); 1 frames decoded; 
  Total: 1 packets (285 bytes) demuxed
Input file #1 (white.png):
  Input stream #1:0 (video): 1 packets read (773 bytes); 1 frames decoded; 
  Total: 1 packets (773 bytes) demuxed
Input file #2 (greyscaleramp.png):
  Input stream #2:0 (video): 1 packets read (790 bytes); 1 frames decoded; 
  Total: 1 packets (790 bytes) demuxed
Output file #0 (ffmpegmaskedmerge_20220302.png):
  Output stream #0:0 (video): 1 frames encoded; 1 packets muxed (1934 bytes); 
  Total: 1 packets (1934 bytes) muxed
3 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0000005d5991c040] Statistics: 285 bytes read, 0 seeks
[AVIOContext @ 0000005d59a1cac0] Statistics: 773 bytes read, 0 seeks
[AVIOContext @ 0000005d59914cc0] Statistics: 790 bytes read, 0 seeks

black.png is RGB 0,0,0
white.png is RGB 255,255,255
greyscaleramp.png is a RGB 0 to 255 ramp, 256x256, where each x position coordinate is equivalent to the R=G=B value

The expected output is the same as greyscaleramp.png , ie. 0-255 perfect gradient

The output value is (-1) than expected after xpos 128. 255 is never reached, so the 2nd input layer is never fully 100% "opaque"

The same trend occur in YUV , using 0,128,128 , 255,128,128, and a full range ramp Y 0 to 255; and the (-1) than expected affects Y only. I did not bother with uploading YUV samples or console output

A different (but probably math related bug) affects overlay with embedded alpha channel as the mask, I will probably post a ticket later

[Elon Musk]

Fixed in dae95b3ffd62ed86cd2e3798c2f281aa67969eca