Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better colour approximation #202

Closed
wants to merge 3 commits into from
Closed

Better colour approximation #202

wants to merge 3 commits into from

Conversation

mina86
Copy link
Contributor

@mina86 mina86 commented Aug 18, 2018

I’ll let you decide whether this is worth it. It almost feels like it should be separate crate or part of ansi_term.

@mina86 mina86 force-pushed the master branch 2 times, most recently from db35958 to d2bc9e0 Compare August 19, 2018 01:49
@sharkdp
Copy link
Owner

sharkdp commented Aug 19, 2018

I’ll let you decide whether this is worth it.

Thank you for putting that much work into it. Are there any noticeable differences?

It almost feels like it should be separate crate or part of ansi_term.

That would be great!

@mina86
Copy link
Contributor Author

mina86 commented Aug 19, 2018

The differences can be observed though they are subtle. To be fair, some are arguably worse (left is original, right is with change):

colours

This could be avoided by not trying greyscale ramp for non-grey colours.

Below is a full list of colours used in each theme. Lines with two columns indicate old and new methods give the same result, three columns show colour in the theme, approximation using old code and as last column approximation using new code:

==== 1337-Scheme/1337.tmTheme ====

#191919  233 #121212  234 #1c1c1c
#3b3a32   58 #5f5f00  237 #3a3a3a
#3d3d3d  237 #3a3a3a
#515151  239 #4e4e4e
#565656  239 #4e4e4e  240 #585858
#6699cc  110 #87afd7   68 #5f87d7
#66d9ef  116 #87d7d7   81 #5fd7ff
#66a9ec  110 #87afd7   75 #5fafff
#6d6d6d  241 #626262  242 #6c6c6c
#75715e  101 #87875f  242 #6c6c6c
#8cdaff  117 #87d7ff
#90e7f7  116 #87d7d7  117 #87d7ff
#97d8ea  116 #87d7d7
#999999  246 #949494
#a6e22e  148 #afd700
#cc00ff  165 #d700ff
#cd5ac5  169 #d75faf  170 #d75fd7
#d66990  174 #d78787  168 #d75f87
#e05d8c  168 #d75f87
#e5a5e0  182 #d7afd7
#f8f8f0  188 #d7d7d7  231 #ffffff
#f8f8f2  188 #d7d7d7  231 #ffffff
#f92672  162 #d70087  197 #ff005f
#fab85a  179 #d7af5f  215 #ffaf5f
#ffb2f9  218 #ffafd7  219 #ffafff
#ffd0fb  224 #ffd7d7  225 #ffd7ff
#ffd2a6  223 #ffd7af
#ffe1fc  224 #ffd7d7  225 #ffd7ff
#d0d0d0  251 #c6c6c6  252 #d0d0d0
#d699ff  183 #d7afff  177 #d787ff
#e9fdac  187 #d7d7af  193 #d7ffaf
#ecfdb9  187 #d7d7af  229 #ffffaf
#f92649  161 #d7005f  197 #ff005f
#fbdfb5  187 #d7d7af  223 #ffd7af
#fbe3bf  187 #d7d7af  223 #ffd7af
#fc9354  173 #d7875f  209 #ff875f
#fdb082  180 #d7af87  216 #ffaf87
#ff5e5e  203 #ff5f5f
#ff8942  209 #ff875f
#ff9664  209 #ff875f
#ffffff  231 #ffffff

==== DarkNeon/DarkNeon.tmTheme ====

#000000   16 #000000
#00a0a0   37 #00afaf
#0e2231   16 #000000  234 #1c1c1c
#212121  234 #1c1c1c
#242424  234 #1c1c1c  235 #262626
#253b22   22 #005f00  236 #303030
#333333  236 #303030
#420e09   52 #5f0000
#494949  238 #444444
#4a410d   58 #5f5f00
#555555  239 #4e4e4e  240 #585858
#562d56   53 #5f005f  237 #3a3a3a
#578bb3   67 #5f87af
#60a633   71 #5faf5f
#62b1fe   74 #5fafd7   75 #5fafff
#632d04   52 #5f0000
#66ccff  117 #87d7ff   81 #5fd7ff
#676767  241 #626262
#751012   88 #870000
#7c7c7c  243 #767676  244 #808080
#8693a5  103 #8787af
#87c38a  108 #87af87  114 #87d787
#8996a8  103 #8787af
#8a9a95  108 #87af87  246 #949494
#8b98ab  103 #8787af
#8f9d6a  108 #87af87  107 #87af5f
#99cc99  151 #afd7af  114 #87d787
#9b5c2e  130 #af5f00
#aaaaaa  247 #9e9e9e  248 #a8a8a8
#afc4db  146 #afafd7  152 #afd7d7
#b18a3d  137 #af875f
#b1b3ba  145 #afafaf  249 #b2b2b2
#c6a24f  143 #afaf5f  179 #d7af5f
#cae2fb  152 #afd7d7  189 #d7d7ff
#ccff66  192 #d7ff87  191 #d7ff5f
#cfcb90  180 #d7af87  186 #d7d787
#dad085  186 #d7d787
#e18964  173 #d7875f
#e1d4b9  187 #d7d7af
#e9c062  179 #d7af5f
#ededed  254 #e4e4e4  255 #eeeeee
#eeeeee  254 #e4e4e4  255 #eeeeee
#f8f8f8  255 #eeeeee  231 #ffffff
#f9ee98  186 #d7d787  228 #ffff87
#fd5ff1  170 #d75fd7  207 #ff5fff
#fedcc5  187 #d7d7af  224 #ffd7d7
#fee09c  187 #d7d7af  223 #ffd7af
#ff73fd  212 #ff87d7  213 #ff87ff
#ff8000  208 #ff8700
#ffd2a7  223 #ffd7af
#ffffb6  229 #ffffaf
#ffffff  231 #ffffff
#aaaaaa  247 #9e9e9e  248 #a8a8a8
#c6c5fe  146 #afafd7  189 #d7d7ff

==== TwoDark/TwoDark.tmTheme ====

#272b33   17 #00005f  235 #262626
#282c34   17 #00005f  235 #262626
#2b313a   17 #00005f  236 #303030
#2d2d2d  235 #262626  236 #303030
#393939  236 #303030  237 #3a3a3a
#3d4350   59 #5f5f5f  238 #444444
#515151  239 #4e4e4e
#528bff   69 #5f87ff
#56b6c2   73 #5fafaf
#5c6370   60 #5f5f87  241 #626262
#61afef   74 #5fafd7   75 #5fafff
#747369  102 #878787  243 #767676
#98c379  108 #87af87  114 #87d787
#abb2bf  145 #afafaf  249 #b2b2b2
#be5046  131 #af5f5f
#c678dd  140 #af87d7  176 #d787d7
#d02000  160 #d70000
#d19a66  180 #d7af87  173 #d7875f
#ddb700  178 #d7af00
#e06c75  174 #d78787  168 #d75f87
#e5c07b  180 #d7af87
#ffffff  231 #ffffff
#a09f93  144 #afaf87  247 #9e9e9e
#d27b53  173 #d7875f
#f2777a  174 #d78787  210 #ff8787
#f99157  173 #d7875f  209 #ff875f

==== github-sublime-theme/GitHub.tmTheme ====

#000000   16 #000000
#003300   22 #005f00
#0086b3   31 #0087af
#0086b3   31 #0087af
#009926   34 #00af00   28 #008700
#183691   24 #005f87
#333333  236 #303030
#445588   60 #5f5f87
#63a35c   71 #5faf5f
#75715e  101 #87875f  242 #6c6c6c
#770000   88 #870000
#795da3   97 #875faf
#969896  102 #878787  246 #949494
#990073  126 #af0087   90 #870087
#a71d5d  125 #af005f
#b0cde7  152 #afd7d7
#ddffdd  194 #d7ffd7
#e0e0e0  252 #d0d0d0  254 #e4e4e4
#ededed  254 #e4e4e4  255 #eeeeee
#f8eec7  187 #d7d7af  230 #ffffd7
#f93232  160 #d70000  203 #ff5f5f
#ffdddd  224 #ffd7d7
#ffe792  222 #ffd787
#ffffff  231 #ffffff
#a71d5d  125 #af005f
#df5000  166 #d75f00
#ececec  254 #e4e4e4  255 #eeeeee

==== sublime-monokai-extended/Monokai Extended Bright.tmTheme ====

#000000   16 #000000
#1c1c1c  233 #121212  234 #1c1c1c
#222222  234 #1c1c1c  235 #262626
#272822   16 #000000  235 #262626
#3b3a32   58 #5f5f00  237 #3a3a3a
#3bc0f0   74 #5fafd7   75 #5fafff
#3e3d32   58 #5f5f00  237 #3a3a3a
#565656  239 #4e4e4e  240 #585858
#66d9ef  116 #87d7d7   81 #5fd7ff
#73817d  102 #878787  244 #808080
#75715e  101 #87875f  242 #6c6c6c
#93a1a1  109 #87afaf  247 #9e9e9e
#965912   94 #875f00
#9d550f  130 #af5f00
#9df39f  151 #afd7af  157 #afffaf
#a6e22e  148 #afd700
#ae81ff  141 #af87ff
#b42a1d  124 #af0000
#c6cece  152 #afd7d7  252 #d0d0d0
#cfcfc2  187 #d7d7af  252 #d0d0d0
#e0eddd  188 #d7d7d7  254 #e4e4e4
#e6db74  186 #d7d787
#ec3533  167 #d75f5f  203 #ff5f5f
#edf080  186 #d7d787  228 #ffff87
#f6aa11  178 #d7af00  214 #ffaf00
#f6f080  186 #d7d787  228 #ffff87
#f8f8f0  188 #d7d7d7  231 #ffffff
#f8f8f2  188 #d7d7d7  231 #ffffff
#f92672  162 #d70087  197 #ff005f
#fc951e  172 #d78700  208 #ff8700
#fd971f  172 #d78700  208 #ff8700
#ff3a28  202 #ff5f00
#ffe792  222 #ffd787
#ffffff  231 #ffffff
#ab6515  130 #af5f00
#bbbbbb  249 #b2b2b2  250 #bcbcbc
#bf7117  136 #af8700  130 #af5f00
#cc4273  168 #d75f87
#d47d19  172 #d78700
#e69f66  180 #d7af87  179 #d7af5f
#e8891c  172 #d78700
#fc951e  172 #d78700  208 #ff8700
#fffff8  230 #ffffd7  231 #ffffff

==== sublime-monokai-extended/Monokai Extended Light.tmTheme ====

#000000   16 #000000
#0089b3   31 #0087af
#06b512   34 #00af00
#073642   23 #005f5f  236 #303030
#3b3a32   58 #5f5f00  237 #3a3a3a
#3bc0f0   74 #5fafd7   75 #5fafff
#49483e   59 #5f5f5f  238 #444444
#565656  239 #4e4e4e  240 #585858
#586e75   66 #5f8787  242 #6c6c6c
#636050   59 #5f5f5f
#666663  101 #87875f  241 #626262
#679c00  106 #87af00   70 #5faf00
#684d99   97 #875faf   60 #5f5f87
#693800   94 #875f00   58 #5f5f00
#75715e  101 #87875f  242 #6c6c6c
#7c7865  101 #87875f  243 #767676
#7d4300   94 #875f00
#839496  102 #878787  246 #949494
#914e00   94 #875f00
#93917d  102 #878787  245 #8a8a8a
#998f2f  136 #af8700  100 #878700
#9f9f66  144 #afaf87  143 #afaf5f
#ae81ff  141 #af87ff
#c2a51c  142 #afaf00
#d02000  160 #d70000
#ddb700  178 #d7af00
#df9400  172 #d78700
#fc951e  172 #d78700  208 #ff8700
#ff0000  196 #ff0000
#ff4a52  203 #ff5f5f
#ffffaa  229 #ffffaf
#a65800  130 #af5f00
#ae81ff  141 #af87ff
#b42a1d  124 #af0000
#ba6300  130 #af5f00
#c8cecc  152 #afd7d7  252 #d0d0d0
#cc4273  168 #d75f87
#ccc9ad  181 #d7afaf  187 #d7d7af
#cf6e00  172 #d78700  166 #d75f00
#cf7000  172 #d78700  166 #d75f00
#d3201f  160 #d70000
#dc322f  160 #d70000  166 #d75f00
#e0eddd  188 #d7d7d7  254 #e4e4e4
#e0fdce  188 #d7d7d7  194 #d7ffd7
#e42e70  162 #d70087  161 #d7005f
#e69f66  180 #d7af87  179 #d7af5f
#e6e3c4  187 #d7d7af  188 #d7d7d7
#ec3533  167 #d75f5f  203 #ff5f5f
#f6aa11  178 #d7af00  214 #ffaf00
#f6f080  186 #d7d787  228 #ffff87
#f9005a  161 #d7005f  197 #ff005f
#fafafa  231 #ffffff
#fc951e  172 #d78700  208 #ff8700
#ff3a28  202 #ff5f00
#ff9117  208 #ff8700
#ffe792  222 #ffd787

==== sublime-monokai-extended/Monokai Extended Origin.tmTheme ====

#000000   16 #000000
#222218   16 #000000  234 #1c1c1c
#222222  234 #1c1c1c  235 #262626
#272822   16 #000000  235 #262626
#3b3a32   58 #5f5f00  237 #3a3a3a
#3e3d32   58 #5f5f00  237 #3a3a3a
#3bc0f0   74 #5fafd7   75 #5fafff
#49483e   59 #5f5f5f  238 #444444
#565656  239 #4e4e4e  240 #585858
#586e75   66 #5f8787  242 #6c6c6c
#636050   59 #5f5f5f
#66d9ef  116 #87d7d7   81 #5fd7ff
#66d9ef  116 #87d7d7   81 #5fd7ff
#75715e  101 #87875f  242 #6c6c6c
#75715e  101 #87875f  242 #6c6c6c
#7c7865  101 #87875f  243 #767676
#965912   94 #875f00
#9d550f  130 #af5f00
#a6e22e  148 #afd700
#ae81ff  141 #af87ff
#cfcfc2  187 #d7d7af  252 #d0d0d0
#e6db74  186 #d7d787
#f8f8f0  188 #d7d7d7  231 #ffffff
#f8f8f2  188 #d7d7d7  231 #ffffff
#f92672  162 #d70087  197 #ff005f
#fc951e  172 #d78700  208 #ff8700
#fd971f  172 #d78700  208 #ff8700
#ffe792  222 #ffd787
#a6e22e  148 #afd700
#ab6515  130 #af5f00
#ae81ff  141 #af87ff
#b42a1d  124 #af0000
#be84ff  141 #af87ff
#bf7117  136 #af8700  130 #af5f00
#cc4273  168 #d75f87
#cfcfc2  187 #d7d7af  252 #d0d0d0
#d3201f  160 #d70000
#d47d19  172 #d78700
#dc322f  160 #d70000  166 #d75f00
#e0eddd  188 #d7d7d7  254 #e4e4e4
#e0fdce  188 #d7d7d7  194 #d7ffd7
#e42e70  162 #d70087  161 #d7005f
#e69f66  180 #d7af87  179 #d7af5f
#e6db74  186 #d7d787
#e8891c  172 #d78700
#ec3533  167 #d75f5f  203 #ff5f5f
#f6aa11  178 #d7af00  214 #ffaf00
#f6f080  186 #d7d787  228 #ffff87
#f8f8f0  188 #d7d7d7  231 #ffffff
#f92672  162 #d70087  197 #ff005f
#fc951e  172 #d78700  208 #ff8700
#fd971f  172 #d78700  208 #ff8700
#ff3a28  202 #ff5f00
#ff9117  208 #ff8700
#ffffff  231 #ffffff

==== sublime-monokai-extended/Monokai Extended.tmTheme ====

#000000   16 #000000
#073642   23 #005f5f  236 #303030
#1c1c1c  233 #121212  234 #1c1c1c
#222222  234 #1c1c1c  235 #262626
#333333  236 #303030
#3b3a32   58 #5f5f00  237 #3a3a3a
#3bc0f0   74 #5fafd7   75 #5fafff
#444444  237 #3a3a3a  238 #444444
#565656  239 #4e4e4e  240 #585858
#586e75   66 #5f8787  242 #6c6c6c
#636050   59 #5f5f5f
#66d9ef  116 #87d7d7   81 #5fd7ff
#66d9ef  116 #87d7d7   81 #5fd7ff
#75715e  101 #87875f  242 #6c6c6c
#777777  242 #6c6c6c  243 #767676
#7c7865  101 #87875f  243 #767676
#839496  102 #878787  246 #949494
#9d550f  130 #af5f00
#a6e22e  148 #afd700
#aff132  148 #afd700  155 #afff5f
#f92672  162 #d70087  197 #ff005f
#fc951e  172 #d78700  208 #ff8700
#a6e22e  148 #afd700
#ae81ff  141 #af87ff
#b42a1d  124 #af0000
#be84ff  141 #af87ff
#c8cecc  152 #afd7d7  252 #d0d0d0
#cc4273  168 #d75f87
#cfcfc2  187 #d7d7af  252 #d0d0d0
#d3201f  160 #d70000
#dc322f  160 #d70000  166 #d75f00
#e0eddd  188 #d7d7d7  254 #e4e4e4
#e0fdce  188 #d7d7d7  194 #d7ffd7
#e42e70  162 #d70087  161 #d7005f
#e69f66  180 #d7af87  179 #d7af5f
#e6db74  186 #d7d787
#ec3533  167 #d75f5f  203 #ff5f5f
#f6aa11  178 #d7af00  214 #ffaf00
#f6f080  186 #d7d787  228 #ffff87
#f8f8f0  188 #d7d7d7  231 #ffffff
#f8f8f2  188 #d7d7d7  231 #ffffff
#f92672  162 #d70087  197 #ff005f
#fc951e  172 #d78700  208 #ff8700
#fd971f  172 #d78700  208 #ff8700
#ff3a28  202 #ff5f00
#ff9117  208 #ff8700
#ffe792  222 #ffd787
#ffffff  231 #ffffff

@mina86
Copy link
Contributor Author

mina86 commented Aug 19, 2018

I’ve updated the code so greyscale ramp is no longer used for non-grey colours.

Add measure of error introduced when mapping 24-bit colours into
8-bit ANSI palette.  Having such measure makes it possible to
benchmark any further changes meant to improve the approximation.
Improve the way 24-bit colours are mapped to 8-bit ANSI palette by
observing that neither grayscale ramp nor 6×6×6 colour cube have
linear staps throughout.

For example, sequence of values in the cube is: 0, 95, 135, 175, 215
and 255.  The first step is 95 colours while the rest are only 40
colours.  Similarly, grayscale ramp (plus black and white) create
a sequence 0, 8, 18, 28, …, 228, 238, 255.  Here, the first step is
8, the next ones are 10 until final one which is 17.

Assuming the mapping is linear causes wrong results.  For example,
colour #1C1C1C is at index 234 but the code incorrectly mapped it
to index 233 (which is 0x121212).

To further improve the approximation, the new code shifts values
to use rounding during division rather than truncating them.  This
leads #070707 to be approximated by #080808 rather than #000000.
The 6×6×6 colour cube includes gray colours.  Black and white are
already used but the cube also includes other greys such as #5f5f5f
or #d7d7d7.  For those colours it’s better to use colour from the
cube rather than from the grayscale ramp.

Change rgb2ansi_grey to try approximation using coulours in either
section in the pallette and choose the best one.

This only affects 20 colours in total.
@sharkdp sharkdp changed the title Bette colour approximation Better colour approximation Aug 20, 2018
@sharkdp
Copy link
Owner

sharkdp commented Aug 20, 2018

I have compared your version with (1) the truecolor version and (2) the old 8-bit approximation and I have to say that I definitely like your version better!

One thing we should look at before integrating this is performance. The as_terminal_escaped function, which uses rgb2ansi, is called for every single colored piece of output. It's probably not the performance bottleneck, but we should definitely check.

@sharkdp
Copy link
Owner

sharkdp commented Aug 20, 2018

(obviously, this would be rather easy to optimize by calling rgb2ansi only once for every possible color in the highlighting theme)

@sharkdp
Copy link
Owner

sharkdp commented Aug 20, 2018

I ran some simple benchmarks and I cannot measure any significant difference in performance (Benchmark #3 is this PR):

Benchmark #1: COLORTERM="truecolor" bat --color=always src/*.rs

  Time (mean ± σ):     130.4 ms ±   2.2 ms    [User: 123.9 ms, System: 6.1 ms]
 
  Range (min … max):   126.8 ms … 135.9 ms
 
Benchmark #2: COLORTERM=""          bat --color=always src/*.rs

  Time (mean ± σ):     129.6 ms ±   2.2 ms    [User: 122.6 ms, System: 6.7 ms]
 
  Range (min … max):   126.0 ms … 134.7 ms
 
Benchmark #3: COLORTERM=""          target/release/bat --color=always src/*.rs

  Time (mean ± σ):     130.5 ms ±   2.6 ms    [User: 124.4 ms, System: 5.8 ms]
 
  Range (min … max):   126.4 ms … 139.5 ms

@mina86
Copy link
Contributor Author

mina86 commented Aug 20, 2018

(obviously, this would be rather easy to optimize by calling rgb2ansi only once for every possible color in the highlighting theme)

The most obvious way to do it would be with a hash map but I wonder if dealing with hash map would actually be slower than recalculating the colour each time. Best if syntect could be made to return some sort of mutable wrappers around highlighting::Style which would have a cashed calculated index.

But either way, the new calculation method doesn’t add that many operations, so I’m not that surprised that performance effects are negligible. If I was maintaining bat my biggest concern would be whether the additional lines of code are worth it.

@sharkdp
Copy link
Owner

sharkdp commented Aug 21, 2018

But either way, the new calculation method doesn’t add that many operations, so I’m not that surprised that performance effects are negligible.

👍

If I was maintaining bat my biggest concern would be whether the additional lines of code are worth it.

Exactly, that's why I think it would be great if this would be part of ansi_term (or a separate crate?) as you suggested. But I'd be okay with keeping it here for now.

I’ve updated the code so greyscale ramp is no longer used for non-grey colours.

I don't quite understand the reason behind this change. Wouldn't it be "optimal" to just always choose the best match in terms of perceived color difference?

@mina86
Copy link
Contributor Author

mina86 commented Aug 21, 2018

I don't quite understand the reason behind this change. Wouldn't it be "optimal" to just always choose the best match in terms of perceived color difference?

I suppose that depends on which terminal in my second screenshot does better job at highlighting syntax, since that’s bat’s ultimate goal. I think colours do it better than shades of grey.

Exactly, that's why I think it would be great if this would be part of ansi_term (or a separate crate?) as you suggested. But I'd be okay with keeping it here for now.

I can certainly do that, but it will take me potentially a few weeks.

@sharkdp
Copy link
Owner

sharkdp commented Aug 21, 2018

I suppose that depends on which terminal in my second screenshot does better job at highlighting syntax, since that’s bat’s ultimate goal. I think colours do it better than shades of grey.

I think this is a problem with the standard "Monokai Extended" color scheme which doesn't really look great for JSON files.

I think the question should rather be which one resembles the actual color scheme better (in terms of perceived color difference) and the right screenshot is closer to the true color (which the color scheme defins) than the left.

The biggest difference seems to be in the color for comments (original: #75715e). The old code gives #87875f and the new one gives #6c6c6c. I just grabbed some old code of mine and computed the CIE76 delta-E distances:

d(#75715e, #87875f) = 14.0
d(#75715e, #6c6c6c) = 11.4

So yours is clearly better. My only question was why you "updated the code so greyscale ramp is no longer used for non-grey colours" because I would always just choose the nearest neighbor with respect to d(·, ·) whether or not is is on the greyscale ramp.

@mina86
Copy link
Contributor Author

mina86 commented Aug 23, 2018

CIE76 delta-E

So old school. ;) There’s now delta_e which does 2000.

But anyway, let’s then close this PR and I’ll prepare the code as a PR against ansi_term or as separate crate.

@mina86 mina86 closed this Aug 23, 2018
@sharkdp
Copy link
Owner

sharkdp commented Aug 23, 2018

So old school. ;) There’s now delta_e which does 2000.

Didn't know that, thanks! 😄

But anyway, let’s then close this PR and I’ll prepare the code as a PR against ansi_term or as separate crate.

Sounds great.

@sharkdp sharkdp mentioned this pull request Jan 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants