High quality magnification filter

[Wyatt Ward]

Summary of the request:
I'd like a hqx filter for FFmpeg - it would be very useful for video game recordings.
The source is here:
​http://web.archive.org/web/20070125113756/http://www.hiend3d.com/hq3x.html

The non - archive.org version of the site does not contain source code or examples. This filter is really amazing, and I hope someone can implement it. Be aware that there is also a hq2x and a hq4x, but I am linking to 3x because it contains the most information on how the filter works.

[Clément Bœsch]

It might be relevant to consider Hyllian' xBR instead (additionally?): ​http://filthypants.blogspot.com/2012/03/xbr-vs-hqx-interpolation-filter.html

Also note that the original HQX filter code is totally insane. It's basically an unrolled interpolation code of all the possible combinations. It can probably fit on a few lines of code.

[Clément Bœsch]

Working on it.

[Wyatt Ward]

I'd love XBR as well. I think I will take a look at it's code and see if I can manage it, but I'd need a deeper understanding of how ffmpeg filters and libavfilter work.

[Clément Bœsch]

So I've been working on hqx recently, and more specifically on hq2x to begin with. I'll try here to detail the progression of my understanding of that algorithm.

hq2x will transform 1 pixel into 2x2 pixels.
hq3x will transform 1 pixel into 3x3 pixels.
hq4x will transform 1 pixel into 4x4 pixels.

hq2x, hq3x and hq4x all use a 3x3 area around the current pixel to determine the interpolation to use.

Attached is a (hopefully correct) representation of one of the 4 pixels to interpolate on hq2x: the top-left (others should be symmetrical). Note: for hq3x, the symmetry will probably be a bit tricky because the output is odd. And for hq4x, we will probably have to make a symmetry with 2x2 pixels...

So on that picture (hq2x/topleft):

On the left (the 3x3 blocks with the blue gradients) is the chosen interpolation when one of the combination on the right is matched. More blue means a more important coefficient, more white means a less important coefficient. Center pixel is the current one.

On the right is the list of all the different combinations triggering that interpolation. When a pixel is red it means there is a visual difference with the current pixel (center). When present, the green lines link 2 pixels which must have a visual difference (in addition to all the other visual diff conditions represented in red).

In various cases we can observe a common pattern in the combinations. For example for the first interpolation (which actually isn't, we just pick the current pixel), we can distinguish the same pattern with some "optional" differences. The representation could be improved to factorize these combinations and mark the optional diff in a different way. Unfortunately, it's not so easy to factorize because of cases like "exclusive optional differences", or "multiple optional differences only".

It might be possible to also swap the representation, but I'm not sure if that would help.

This picture is generated using ​https://github.com/ubitux/hqx. The repository will continue to evolve while I progress on this. The final goal being of course to try to determine the generic rules triggering these interpolation based on the combinations we observe here.

(BTW, if anyone wants to help, feel free to contact me)

[Clément Bœsch]

Representation for optional combinations implemented (that was painful... code a bit ugly). So now, we have:

• red: must have a diff with center pixel
• gray: must not have a diff with center pixel
• black: whether there is a diff or not doesn't matter

I hope I didn't get it wrong. I think I'll try to implement from this, unless I see a way of improving even more the representation.

(Note: the wrong colors for the coeff intensity are also fixed)

[Clément Bœsch]

Patch for hq2x available on the mailing-list: https://ffmpeg.org/pipermail/ffmpeg-devel/2014-June/158612.html

[Clément Bœsch]

Thread updated including hq4x. hq3x soon to come.

[Clément Bœsch]

hqx filter implemented in

commit ded3c9fd32afe4174b65d5bae843cea9149c8f11
Author: Clément Bœsch <u@pkh.me>
Date:   Fri Jun 7 10:57:29 2013 +0200

    avfilter: add hqx filter (hq2x, hq3x, hq4x)
    
    Partially fixes Ticket #3404 (xbr filter remaining)

@wyatt8740: do you plan to work on xbr?

[Clément Bœsch]

An XBR patch for x2 scaling is currently under review @ ​http://ffmpeg.org/pipermail/ffmpeg-devel/2014-October/164464.html

[Clément Bœsch]

xBR x2 x3 x4 implemented in 19d0949d31b0d2db4d01a667d515ce5099cb0530 by Arwa Arif.