Sixel quality improvements

[hpjansson]

Discussed in #173

^{Originally posted by helma October 4, 2023}
I intend to use chafa as a previewer for my photos, but have sometimes problems with the rendering quality of slow color gradients (e.g. cloudy skies), that come out very quantized with distinct color bands. Crisp photos render however perfectly as sixels. I have tried chafa, notcurses, img2sixel and imagemagick and in general I can get the best results with the -I option of img2sixel, followed by magick, notcurses and chafa (my terminal is foot version: 1.14.0 +pgo +ime +graphemes -assertions from nixos 23.05). Here are the resulting sixel files, png images have been generated with sixel2img:

Original (not very beautiful, but shows the effect very clearly):

chafa source.jpeg > chafa.six (Chafa version 1.12.4)

https://files.pdp8.info/chafa.six

ncplayer -t0 -q -k -s scalehi source.jpeg > notcurses.six (ncplayer version 3.0.9)

https://files.pdp8.info/notcurses.six

magick source.jpeg -resize 1914x1188> sixel:- > magick.six (Version: ImageMagick 7.1.1-18 Q16-HDRI x86_64 8c6cdf58e:20230923 https://imagemagick.org)

https://files.pdp8.info/magick.six

img2sixel -I -w 1914 source.jpeg > img2sixel.six (img2sixel 1.10.3)

Conversion to png spoils it completely, but img2sixel gives by far the best approximation of the original with the -I option.
The high color option may cause however artefacts at the bottom of some images (maybe related to libsixel/libsixel#16).

https://files.pdp8.info/img2sixel.six

Are there any options in chafa to improve sixel rendering of photos? Dithering can improve the appearance of this file, but I loose details in sharper shots. I would prefer to use chafa over the other options because of speed and active maintainance.

[hpjansson]

As mentioned in the discussion, there's currently no way for the user to get better quality.

Let's use this issue for improvements to the default quantizer and ways to trade off work/quality with the -w option.

[johnd0e]

we're probably allocating a disproportionate amount of palette entries to the most common color ranges (cf. the dark areas), and too few to rarer colors.

Ideally, the palette should not be the same for every image, but rather tailored to each one.

[hpjansson]

Ideally, the palette should not be the same for every image, but rather tailored to each one.

It does use a dynamic palette, but the palette generator isn't great for quality. I think it's possible to get better results without making performance suffer too much, though.

[jerch]

Wow that cloud image is an interesting one, it also steps into an edge case in my octree impl cutting off too many of the "second line colors" during my 2nd-pass tree pruning. Thx for that image, a good test case indeed.

[hpjansson]

I spent last weekend reimplementing the palette generator. It's much better now. I did some benchmarks using a new test script I wrote:

Converters
----------

chafa @ 9ed690fc7d905dcc5497261898a31470860b79ef
chafa-old @ 70d1868e98b93f65fd5d05c969962d377961fb5a
magick @ 7.1.1-26 Q16-HDRI
img2sixel @ 1.10.3
ncplayer @ 3.0.9

Invocation
----------

chafa -f sixel --view-size 9999x9999 --exact-size on in.qif > out.six
magick convert +dither in.qif sixel:- > out.six
img2sixel -q full -d none in.qif > out.six
ncplayer -k -t0 -q in.qif > out.six  # In mlterm @ 4k fullscreen

Wrapper: /usr/bin/time -f '%U %e %M'

Notes
-----

chafa output was cropped to remove cell padding along the rightmost
and bottom edges. Nine different specimens were produced for the
new version reflecting the quality range of the -w switch. The old
version only supports a single quality setting.

ncplayer produced output dimensions smaller than the others
(3820x2094 < 3873x2582) and with an incorrect aspect ratio. It was
stretched using nearest-neighbor interpolation to match the common
size exactly while preserving the color palette. The smaller output
lowered its utime, elapsed and peakrss, and also left the quality
measurements inaccurate relative to a hypothetical correctly
proportioned output. Still, it's probably in the correct ballpark.

Dithering was turned off in all tests.

utime, elapsed and peakrss come from GNU time's %U, %e, %M formatting
options, respectively. The tests were run on x86_64 (Haswell i7) with
four physical cores.

Results
-------

specimen ncolors    MSE  SSIM   PSNR  PAE utime elapsed peakrss
original   37568      -     -      -    -     -       -       -
prepared   34683      -     -      -    -     -       -       -
magick       251  2.873 0.967 43.582 5654  3.89    3.96  283296
img2sixel    250  6.857 0.912 39.803 5397  7.90    7.93  114412
ncplayer     294 21.944 0.858 34.752 8995  0.67    0.54  204612
chafa-old    162  6.002 0.964 40.382 8481  0.50    0.15   95104
chafa-w1      24 27.572 0.838 33.760 6939  0.25    0.10   95104
chafa-w2      25 26.801 0.838 33.883 6682  0.26    0.10   94976
chafa-w3     135  7.390 0.929 39.478 4112  0.33    0.11   94984
chafa-w4     143  7.191 0.929 39.597 4112  0.34    0.11   95080
chafa-w5     154  7.221 0.929 39.578 3598  0.34    0.11   95000
chafa-w6     255  3.960 0.949 42.188 4369  0.43    0.13   94984
chafa-w7     255  3.991 0.949 42.154 3598  0.40    0.13   94860
chafa-w8     255  3.976 0.949 42.171 3855  0.42    0.16   94984
chafa-w9     255  3.965 0.949 42.182 3341  0.45    0.16   99008

There's a lot of interesting information in that table. For instance, although the nominal palette size is ~255 colors, different approaches can end up producing a lot more or a lot fewer colors, and quality metrics don't always correlate well with subjective quality.

[hpjansson]

In these tests, it's 24x as fast as IM and 3-4x as fast as ncplayer while producing top-tier output quality.

[hpjansson]

Full image produced by chafa -w 9.

[dankamongmen]

In these tests, it's 24x as fast as IM and 3-4x as fast as ncplayer while producing top-tier output quality.

very nice!

[hpjansson]

@dankamongmen By the way - is there a way to invoke the Notcurses engine so that it replicates the input image dimensions precisely, and without doing a detection round-trip to the terminal? It'd make it much easier to run in a benchmarking harness and get fair results. I had to include it because it's one of the few well-performing implementations, but was in doubt because of the manual tweaking involved (plus I think some of the wall-clock time is spent waiting for the terminal to respond to queries).