"palettegen" adds 1 color on transparent input

[RedError]

Summary of the bug:
​palettegen from a transparent gif, png, adds the invisible transparency color to the palette as a visible color, with no relation to the one selected with the transparency_color option.

On transparent PNG, it will use the rgb value of the transparent areas.
On transparent GIF, it will always add white.

None of the available options seem to change that.

How to reproduce:

ffmpeg -i "transparent.gif" -vf palettegen "palette.png" -y

Of course, using that palette with paletteuse will use the added color in the result, as paletteuse has no way of knowing that it was actually a transparent color.

ffmpeg -i "target.png" -i "palette.png" -lavfi paletteuse=dither=0 "target_merged.gif"

This will randomly add one color where there shouldn't be one.

[RedError]

Examples: (files joint in zip)
Transparent Gif 3 red colors + green transparency
Resulting palette: White in fourth position (because white is bigger than red). Transparent pixel in position 256.

[Parsed_palettegen_0 @ 000001f3f6d11b80] 
4(+1) colors generated out of 4 colors; ratio=1.000000

Transparent Png 3 red colors + green transparency
Resulting palette: Green in second position (because green is smaller than red). Transparent pixel in position 256.

[Parsed_palettegen_0 @ 0000028cb16ed1c0] 
4(+1) colors generated out of 4 colors; ratio=1.000000

Expected result:
Only 3 colors (black, brown, red), transparent pixel in position 256.

---

Using generated palettes on this image:

Palette from gif: white color has been added

Palette from png: green color has been added

Expected result: only red, black, transparency.

[RedError]

Testing files and scripts and reports

[RedError]

Report when creating palette from transparent gif

ffmpeg started on 2020-03-29 at 21:07:34
Report written to "ffmpeg-20200329-210734.log"
Command line:
ffmpeg -i "C:\\Users\\Akira Baes\\Desktop\\Old Anims\\_Version36B\\tp_black_gif.gif" -vf palettegen "C:\\Users\\Akira Baes\\Desktop\\Old Anims\\_Version36B\\\\tp_black_gif pal.png" -y -v 9 -loglevel 99 -report
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.1 (GCC) 20190807
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Splitting the commandline.
Reading option '-i' ... matched as input url with argument 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'palettegen'.
Reading option 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png' ... matched as output url.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument 9.
Applying option loglevel (set logging level) with argument 99.
Applying option report (generate a report) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif.
Successfully parsed a group of options.
Opening an input file: C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif.
[NULL @ 000001998b6f9bc0] Opening 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif' for reading
[file @ 000001998b6facc0] Setting default whitelist 'file,crypto'
[gif @ 000001998b6f9bc0] Format gif probed with size=2048 and score=100
[gif @ 000001998b6f9bc0] Before avformat_find_stream_info() pos: 0 bytes read:1070 seeks:0 nb_streams:1
[gif @ 000001998b6f9bc0] After avformat_find_stream_info() pos: 1070 bytes read:1070 seeks:0 frames:1
Input #0, gif, from 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif':
  Duration: 00:00:00.10, start: 0.000000, bitrate: 85 kb/s
    Stream #0:0, 1, 1/100: Video: gif, 1 reference frame, bgra, 48x48, 0/1, 10 tbr, 100 tbn, 100 tbc
Successfully opened the file.
Parsing a group of options: output url C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png.
Applying option vf (set video filters) with argument palettegen.
Successfully parsed a group of options.
Opening an output file: C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png.
Successfully opened the file.
Stream mapping:
  Stream #0:0 -> #0:0 (gif (native) -> png (native))
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[gif @ 000001998b70e340] code=21 '!'
[gif @ 000001998b70e340] code=2c ','
detected 12 logical cores
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'video_size' to value '48x48'
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'pix_fmt' to value '28'
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'time_base' to value '1/100'
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 000001998b716380] Setting 'frame_rate' to value '100/10'
[graph 0 input from stream 0:0 @ 000001998b716380] w:48 h:48 pixfmt:bgra tb:1/100 fr:100/10 sar:0/1 sws_param:flags=2
[format @ 000001998b798f80] Setting 'pix_fmts' to value 'rgb24|rgba|rgb48be|rgba64be|pal8|gray|ya8|gray16be|ya16be|monob'
[auto_scaler_0 @ 000001998b799100] Setting 'flags' to value 'bicubic'
[auto_scaler_0 @ 000001998b799100] w:iw h:ih flags:'bicubic' interl:0
[format @ 000001998b798f80] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_palettegen_0' and the filter 'format'
[AVFilterGraph @ 000001998b70e080] query_formats: 4 queried, 2 merged, 1 already done, 0 delayed
[auto_scaler_0 @ 000001998b799100] picking rgba out of 10 ref:bgra alpha:1
[swscaler @ 000001998b79d640] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scaler_0 @ 000001998b799100] w:16 h:16 fmt:bgra sar:1/1 -> w:16 h:16 fmt:rgba sar:1/1 flags:0x4
Output #0, image2, to 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0, 0, 1/10: Video: png, 1 reference frame, rgba, 16x16 [SAR 1:1 DAR 1:1], 0/1, q=2-31, 200 kb/s, 10 fps, 10 tbn, 10 tbc
    Metadata:
      encoder         : Lavc58.54.100 png
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[Parsed_palettegen_0 @ 000001998b716d80] 4(+1) colors generated out of 4 colors; ratio=1.000000
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
Clipping frame in rate conversion by 0.000008
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[out_0_0 @ 000001998b7028c0] EOF on sink link out_0_0:default.
No more output streams to write to, finishing.
[image2 @ 000001998b70e7c0] Opening 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png' for writing
[file @ 000001998b79c380] Setting default whitelist 'file,crypto'
[AVIOContext @ 000001998bed4000] Statistics: 0 seeks, 1 writeouts
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=3.51x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_gif.gif):
  Input stream #0:0 (video): 1 packets read (1069 bytes); 1 frames decoded; 
  Total: 1 packets (1069 bytes) demuxed
Output file #0 (C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_gif pal.png):
  Output stream #0:0 (video): 1 frames encoded; 1 packets muxed (113 bytes); 
  Total: 1 packets (113 bytes) muxed
1 frames successfully decoded, 0 decoding errors
[AVIOContext @ 000001998b702f80] Statistics: 1070 bytes read, 0 seeks

Report when creating palette from transparent png

ffmpeg started on 2020-03-29 at 21:01:38
Report written to "ffmpeg-20200329-210138.log"
Command line:
ffmpeg -i "C:\\Users\\Akira Baes\\Desktop\\Old Anims\\_Version36B\\tp_black_png.png" -vf palettegen "C:\\Users\\Akira Baes\\Desktop\\Old Anims\\_Version36B\\\\tp_black_png pal.png" -y -v 9 -loglevel 99 -report
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.1 (GCC) 20190807
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Splitting the commandline.
Reading option '-i' ... matched as input url with argument 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'palettegen'.
Reading option 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png' ... matched as output url.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument 9.
Applying option loglevel (set logging level) with argument 99.
Applying option report (generate a report) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png.
Successfully parsed a group of options.
Opening an input file: C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png.
[NULL @ 0000028cb16b9bc0] Opening 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png' for reading
[file @ 0000028cb16bacc0] Setting default whitelist 'file,crypto'
[png_pipe @ 0000028cb16b9bc0] Format png_pipe probed with size=2048 and score=99
[png_pipe @ 0000028cb16b9bc0] Before avformat_find_stream_info() pos: 0 bytes read:1418 seeks:0 nb_streams:1
[png_pipe @ 0000028cb16b9bc0] After avformat_find_stream_info() pos: 1418 bytes read:1418 seeks:0 frames:1
Input #0, png_pipe, from 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png':
  Duration: N/A, bitrate: N/A
    Stream #0:0, 1, 1/25: Video: png, 1 reference frame, pal8(pc), 48x48, 0/1, 25 tbr, 25 tbn, 25 tbc
Successfully opened the file.
Parsing a group of options: output url C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png.
Applying option vf (set video filters) with argument palettegen.
Successfully parsed a group of options.
Opening an output file: C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png.
Successfully opened the file.
detected 12 logical cores
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> png (native))
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'video_size' to value '48x48'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'pix_fmt' to value '11'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'time_base' to value '1/25'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0000028cb16eda40] w:48 h:48 pixfmt:pal8 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
[format @ 0000028cb177b6c0] Setting 'pix_fmts' to value 'rgb24|rgba|rgb48be|rgba64be|pal8|gray|ya8|gray16be|ya16be|monob'
[auto_scaler_0 @ 0000028cb16f3040] Setting 'flags' to value 'bicubic'
[auto_scaler_0 @ 0000028cb16f3040] w:iw h:ih flags:'bicubic' interl:0
[Parsed_palettegen_0 @ 0000028cb16ed1c0] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_palettegen_0'
[auto_scaler_1 @ 0000028cb16f4d40] Setting 'flags' to value 'bicubic'
[auto_scaler_1 @ 0000028cb16f4d40] w:iw h:ih flags:'bicubic' interl:0
[format @ 0000028cb177b6c0] auto-inserting filter 'auto_scaler_1' between the filter 'Parsed_palettegen_0' and the filter 'format'
[AVFilterGraph @ 0000028cb16ecd40] query_formats: 4 queried, 1 merged, 2 already done, 0 delayed
[auto_scaler_1 @ 0000028cb16f4d40] picking rgba out of 10 ref:bgra alpha:1
[swscaler @ 0000028cb177ed80] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scaler_0 @ 0000028cb16f3040] w:48 h:48 fmt:pal8 sar:0/1 -> w:48 h:48 fmt:bgra sar:0/1 flags:0x4
[swscaler @ 0000028cb1ad64c0] Forcing full internal H chroma due to input having non subsampled chroma
[auto_scaler_1 @ 0000028cb16f4d40] w:16 h:16 fmt:bgra sar:1/1 -> w:16 h:16 fmt:rgba sar:1/1 flags:0x4
Output #0, image2, to 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0, 0, 1/25: Video: png, 1 reference frame, rgba, 16x16 [SAR 1:1 DAR 1:1], 0/1, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.54.100 png
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[Parsed_palettegen_0 @ 0000028cb16ed1c0] 4(+1) colors generated out of 4 colors; ratio=1.000000
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
Clipping frame in rate conversion by 0.000008
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[out_0_0 @ 0000028cb16ee6c0] EOF on sink link out_0_0:default.
No more output streams to write to, finishing.
[image2 @ 0000028cb16cd040] Opening 'C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png' for writing
[file @ 0000028cb16f4440] Setting default whitelist 'file,crypto'
[AVIOContext @ 0000028cb1cc4800] Statistics: 0 seeks, 1 writeouts
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=1.19x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\tp_black_png.png):
  Input stream #0:0 (video): 1 packets read (1418 bytes); 1 frames decoded; 
  Total: 1 packets (1418 bytes) demuxed
Output file #0 (C:\Users\Akira Baes\Desktop\Old Anims\_Version36B\\tp_black_png pal.png):
  Output stream #0:0 (video): 1 frames encoded; 1 packets muxed (113 bytes); 
  Total: 1 packets (113 bytes) muxed
1 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0000028cb16c2f80] Statistics: 1418 bytes read, 0 seeks

[RedError]

Extra experiments:
Transparent Gif with white color added
Same palette as normal gif
It will however mention that one color was a dupe:

[Parsed_palettegen_0 @ 000002179de81c80] 
5(+1) colors generated out of 5 colors; ratio=1.000000
[Parsed_palettegen_0 @ 000002179de81c80] 
Dupped color: FFFFFFFF

Animated transparent gif with different transparency color on each frame
Same palette as non-animated gif
Same output as non-animated version

[Parsed_palettegen_0 @ 000001ec2c9e6e00] 
4(+1) colors generated out of 4 colors; ratio=1.000000

Transparent PNG with several colors hidden with alpha zero
Palette with all the invisible colors added

[Parsed_palettegen_0 @ 0000017f22529b00] 
8(+1) colors generated out of 8 colors; ratio=1.000000

[Carl Eugen Hoyos]

I don't think this can be changed.

[RedError]

Replying to cehoyos:

I don't think this can be changed.

The change would be to ignore transparent colors when creating the palette.
I wrote something similar in PIL, I don't see why this would not be able to be changed in ffmpeg.

Or is there something I'm missing? I'm not sure why this has to be closed.

[RedError]

The fact that with the "Transparent Gif with white color added" experiment it does recognise that there are two different whites (duplicate detected), meaning that the histogram didn't recognise it as the same color (different hashes), is an indication that there is a way to distinguish a color from the image and a ghost color, at least for GIF.

It's here:
​https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L459

const uint32_t *p = (const uint32_t *)(f->data[0] + y*f->linesize[0]);

        for (x = 0; x < f->width; x++) {
            ret = color_inc(hist, p[x]);

The solution is to not call color_inc if p[x]'s alpha is zero.

Alpha is present all the way up the histogram's boxes. It's only stripped away at the end here: ​https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L186

[Carl Eugen Hoyos]

But the filter is not gif-only.

[RedError]

Replying to cehoyos:

But the filter is not gif-only.

And? The gif discussion was only how I got to that conclusion. The fix doesn't make any assumption about the input being a gif.

The solution is to not call color_inc if p[x]'s alpha is zero.

This applies to PNG and GIF and any format that has Alpha/transparency.
For any of those, if the color has an alpha of Zero, the user doesn't expect it to appear in the palette.

[MasterQuestionable]

I don't quite understand the problem.
Is it still relevant now?