[Discuss] tradeoffs on the single-palette guidance for data visualization

[monfera]

tl;dr

• category color readability and discernibility may be more important than constant RGB for a categorical value
• color perception depends a lot on shape and size, and it has direct datavis consequences
• therefore, we already don't have uniform category colors in the eye of the viewer (plus there's thinning in outer sunburst rings)
• color thinning for partitioning charts would achieve goals in datavis (more uniform perception and better readability of color), accessibility (higher contrast) and aesthetics (more appealing treemap/sunburst/...)
• proposed step: use the color-under-text palette for partitioning charts

(from Maureen Stone's In Color Perception, Size Matters)

---

When charts share categories, sharing the value->color assignment helps the reader:

• the user immediately connects the color with the category, and looking at the second chart will already be with this knowledge
• it'll also be clear that some charts indeed share the same categorical dimension
• easier to compare values across charts for the same category - eye movement to the same color
• fewer eye movements between chart proper and legend swatch
  

This comment, rightly, refers to this need. Initially it seems like a blocker to possibly using lighter, less saturated colors for partitioning charts that have obligate (treemap) or preferred (pie/sunburst) color backgrounds underneath text.

The two aspects - color unity, and balanced color use - are hard to reconcile on a physical, ie. wavelength / RGB basis, but they seem reconcilable perceptually.

As color sensation heavily depends on geometry and surroundings eg. background color, there are two kinds of issues when sticking to the same exact RGB values:

1. Within a given chart: with ever smaller geometries, it's getting harder to distinguish among colors; conversely, with increasing size, chroma and darkness become overbearing
2. Across charts: the same RGB value will look different between eg. a time series chart (thin lines) and a treemap (large areas)

So a RGB color that's OK for a line chart will be a bit too saturated and dark (assuming light background) on a treemap.

There's a variety of results in color discernibility for data visualization purposes.

Danielle Szafir suggested research by Maureen Stone, which doesn't suggest an automated way, yet describes the approaches of (briefly)

• optimizing a color palette for a good balance between small and large shapes - our colorblind palette is designed with this information already, as it got lightened/redesigned when partitioning charts with larger ink areas and text overlays were added
• allow tweaking eg. lightening/boosting, as even with a balanced singlular palette, there'll be big differences in discernibility and sensation of chroma

...color appearance changes dramatically with the size of the object being viewed
...the strongest colors are darker and more colorful than are ideal for display behind text
When designing colors for very large regions, the challenge is to keep the color from being too dark and too colorful, especially when it is used behind other visual elements, such as text...

We already have precedent for color thinning, as the outer rings of partitioning charts as are in the storybook examples

Other aspects:

• Accessibility is improved with increased text to background contrast; WCAG constraints are just that (also, some CIELAB assumptions are oversimplifying with reference to perceptual properties)
• Aesthetics is also important, and the baseline colorblind safe palette is not just dominating but unappealing (imo) with the larger surface of partitioning charts

Action:

• Design feedback on whether we can test a simple dual palette approach, ie.
  • baseline palette for Cartesian plots and legends, and
  • corresponding colors of the ink-under-text palette for partitioning charts, while also retaining the lightening gradient on sunburst charts. Also, perhaps TSVB should offer color/chroma lightening or boosting options before offering individual color picking
• looping in Product, as it's about data visualization colors for a (in good part) data visualization solution
• looping in Maps; much lower likelihood of using the same categorical color assignment for eg. choropleth map (quantitative palette) vs small map markers (typically, categorical palette), but might still be interesting generally, due to large size differences
• in the future, we might consider slightly higher sophistication than just switching according to chart type, eg. bubble charts could have wildly differing circle sizes, and less often, there are very very thin bars too
• assuming we'll work with lighter palettes at some point, let's consider legend and color picker boosting, mentioned here: "this made the small samples in the color picker difficult to distinguish. To solve this problem, we systematically increased the chroma of the samples until they were visually distinct"

[elizabetdev]

I had a zoom call with @monfera to try to understand this issue.

And the idea is to support the use of the euiPaletteColorBlindBehindText rather than the euiPaletteColorBlind() for partitioning charts. The euiPaletteColorBlind() would be used for cartesian plots and legends.

As we can see, when we have situations like the following one (legends on top) the euiPaletteColorBlindBehindText looks better (top sunburst). But when it's used too many steps it gets white.

I agree that the euiPaletteColorBlindBehindText looks better for these situations. But maybe there are a few adjustments to make to avoid getting to white. But making these adjustments would probably lead to a11y issues. But also dealing with a lot of colors is hard. 🤷‍♀️

@cchaos what do you think?

[cchaos]

I agree that the behind text variant is better suited for charts that have text directly on top of the color. Hence why the palette itself was created. 😉

The decision point is:

Dashboards using the same "color palette" for all charts and using the same color for the same series across those charts.

If you use euiPaletteColorBlind for x/y charts and euiPaletteColorBlindBehindText parts-to-whole charts, you are using variants of the same palette but not the actual same palette. For example, Lens right does do this where the Pie charts use the behind text variant but bar charts do not:

To those with some sort of eye sight deficiency would the differences between the green variants actually indicate to them different series? They look similar but they're not.

What we cannot do is change the bar charts to use the behind-text variant because the contrast levels for graphics clearly states that there needs to be a 3.0 ratio which the behind text variants do not. In fact, I think even some of our regular color blind palette colors aren't quite at 3.0 either.

[monfera]

Thanks @miukimiu and @cchaos for having spent work on it. You're right, it doesn't seem doable if the area chart and histogram bar chart can't use euiPaletteColorBlindBehindText or other lightening due to WCAG constraints.

Legends look solvable: bars and pie/treemap could both use euiPaletteColorBlind in the legend, as it's a good match for the sunburst too - not the same RGB, instead, close apparent hue and value as our small swatch benefits from the relative boosting (users won't measure RGB; see boosting at Fig. 6)

To match the euiPaletteColorBlindBehindText pie/treemap, the area charts and histogram bar charts should also use lightening eg. euiPaletteColorBlindBehindText but according to your info Caroline it's not feasible, because some colors would be too light.

I'd need to learn to avoid false tracks - is this a requirement per categorical color in dataviz, or only for general UI? Tableau is using some colors lighter than in euiPaletteColorBlind, at least as light as in euiPaletteColorBlindBehindText :

Quantitative palettes have light ranges too, maybe contrast rules are different for them:

We have quantitative palettes with light colors too.

With only two categories, Tableau's colors are more discernible, whether viewed in color or grayscale:

Moving the goalpost further still, we could eventually increase discernibility by front-loading the category palette:

• having 2 - 4 categories is fairly common
• the dashboard maker may group/bin into 2 - 4 categories with the express intent of making it more discernible, and larger differences among the first few palette colors, and avoiding close adjacent colors/values (light-dark-light-dark) would help too, as most charts put adjacent colors side by side
• ordering close to the color wheel order reduces sequential discernibility and not always as pleasing to look at, eg. green/blue

[monfera]

...@flash1293 I may have read too much into the colorwheel order in the style guide - is Lens / Kibana at liberty of picking colors in a different order?

[flash1293]

@monfera As there is an abstraction layer between the EUI code and the Lens chart we could put this kind of logic there, but should we? Is there a reason the palette shouldn't be applied in the same order everywhere?

[monfera]

@flash1293 I didn't mean to limit the question to Lens only; if Kibana, and Lens within it currently picks colors from the euiPaletteColorBlind or euiPaletteColorBlindBehindText palette sequentially, ie. in this order:

then the discernibility between adjacent picks is reduced as it follows a rough color wheel order, and the lightness isn't always too different either. For example, the first pick is green, the second pick is blue, which is quite close to green in terms of both chroma and lightness, see the previous comment. Most of the adjacent colors are fairly close. Most charts that pick colors, eg. pie, stacked bar chart etc. put subsequently picked colors side by side. The effect is, optimization against contrasty neighbor colors.

There are some color palettes that also assume that users will pick colors from the palette in sequence, but they introduce light/dark striping to be more contrasty in lightness eg. ColorBrewer

[flash1293]

@monfera Makes sense, sounds like we should change the order of euiPaletteColorBlind upstream in EUI and Lens (as well as other consumers) will pick it up.

[cchaos]

The main reason we haven't changed the main order of euiPaletteColorBlind() is because it would be a major breaking change to consumers who rely on the particular order of those colors to match representations of series they've already established. (Thinking APM.) I do agree that changing the order of the colors to result in more contrast between neighbors would be ideal, we just have to tread carefully and ensure that we're not detrimentally breaking someone's defined order.

I'd be careful, though, using Color Brewer as "good" examples. They have some interesting advice, but I'm not sure they've actually tested their palettes. For instance the palette above that you presented does not pass any of the color-blind simulations.

Use this tool for palette checking: https://gka.github.io/palettes
And this is the URL for that screenshot above (youll have to copy/paste the url) https://gka.github.io/palettes/#/10|s|a6cee3,1f78b4,b2df8a,33a02c,fb9a99,e31a1c,fdbf6f,ff7f00,cab2d6,6a3d9a|ffffe0,ff005e,93003a|0|0

Though as soon as you start tinting any of the original palette colors, you'll lose that 3.0 contrast against the background, though you would gain better contrast ratios to siblings. It's a really balancing act.

Some links to read more:

• https://webaim.org/articles/contrast/#sc1411
• https://www.w3.org/WAI/GL/low-vision-a11y-tf/wiki/Informational_Graphic_Contrast_(Minimum)

All this said, if we had graph alternatives - captions, screen reader descriptions, tables, other kinds of alt-text to describe the charts across the board then we wouldn't have to rely so heavily on ensuring contrast minimums. But we dont.

[cchaos]

Also, here's the Issue where I discuss in great detail the latest evolution to the current palette #2238

[monfera]

Thanks @cchaos for your comments and links. The more details you share, the more tradeoffs I learn about. I value the ton of work and insight you put into palette iterations, doing it along all the other workstreams you've simultaneously run 🙇

• Good point about backwards compatibility. New dashboards esp. with Lens, treemap not yet in GA may not need to be constrained by past color assignments, but it'd add a migration concern—current dashboards getting an "use old sequence" flag, or new ones a "new sequence" flag. Though it's just a subset of dashboards where users care and categories don't change, we can't know. Support for manual color assignments would allow users lock in historic colors.
• Brewer's site also marks it as non-colorblind safe, it's just an example for lightness steps on subsequent colors, I respect the accessibility work that went into euiPaletteColorBlind as a 10-cat colorblind safe palette is a big challenge even with no legacy and other guardrails
• Fair point on the weakness of other accessibility avenues, thanks!

There seems to be some shared feeling on the issues, yet no low hanging fruit for an improvement. The pie/treemap cells and bars are not as appealing and lightness-discernible as they could be, I respect past design iterations, conflicting goals (eg. Tableau palette extends well into darker colors while ours doesn't) and much of the backward compatibility angle. I tried to raise things that had been considered, thanks again for taking the time to answer and link.

[myasonik]

Just to throw something out there in case it hasn't been considered (haven't seen it mentioned here yet).

If there aren't enough colors to go around for everything, shading can also be used to differentiate segments. This isn't a solution for every chart or dashboard but could be something that apps could opt into themselves without having to force a change into EUI.

(By shading, I mean "striped" vs "solid" vs "dotted" as examples.)

I don't think there's a single silver bullet to this problem that will solve all the problems, so y'all might need to take multiple strategies and apply them individually in specific areas.

[monfera]

@myasonik indeed, patterns are great (enlisted eg. here) while they have some limitations and optical illusions or interference with the overall shape can be a concern. There are related redundant encodings eg. markers of different shape per line like here, also not without drawbacks. The link has calls for avoiding overuse of color; going round and round yielding tens of colors/patterns won't be very readable anyway 😃

[flash1293]

In this case we can also do the re-ordering as part of the Kibana side charts plugin to not break BWC for other consumers.

Did we come to a conclusion about how to treat *behindText with partition charts? If both ways (using regular color blind palette for everything vs using behind text version for partition charts) have drawbacks, which way do we pick?

It's probably not that easy, but should we introduce a new palette that works behind text but is visually closer to the regular color blind palette to use for partition charts?

I think we leaned towards simply using the regular palette for everything on the original PR.

[monfera]

@flash1293 yes it appears so. the WCAG constraint Caroline linked unfortunately doesn't distinguish between small/thin shapes and shapes with larger areas, so I agree with Caroline's conclusion that we can't simply use the euiPaletteColorBlindBehindText palette for partitioning charts and call it a day.

We could have more complicated logic, eg.

1. use euiPaletteColorBlindBehindText for partitioning charts for treemaps, as well as those pies/sunbursts where the labels are placed on the slices, or linked with a callout to the slice (which is the preferred thing to do anyway) but use the darker palette if there are no pie labels and the viewer is forced to rely on color coding against the legend
2. use euiPaletteColorBlindBehindText for the large areas of area charts assuming accessibility feature X is also available
3. ... but allow the user to manually commit to one or the other
4. round robin in current, rough colorwheel order unless the dashboard is new, OR the user set an enabling flag in an old dashboard

and I think, over time, we should, as well as we should improve on other aspects of accessibility, but I'm not in a decision making position on this.

Same thing with the color order of the palette: Caroline points out the legit concern of backwards compatibility with formerly created dashboards. Which is solvable by a legacy flag. But it still represents work and added complexity.

So my concerns and the lack of solution for them remain, but I've the info to say that it's not a low hanging fruit topic and Design put in a lot of legwork to get us to where we are.

So, the question is, what to do short term vs. long term.

Short term: there seems to be no Product or Design initiated project or resource allocated to improving the matter. And it's not low hanging fruit as Design has put a lot of work in color already.

Longer term, we could

• figure out participants, I suggest at least Design, Product, Datavis, KibanaApp, Maps, while looping in specific areas eg. ML, APM
• think about how we want to address the color saliency, esp. the color <=> geometry size relationship - what's good coloring for thin lines is not good for treemaps and area chart, and the other way around
• eg. come up with a dual-palette approach that picks one or the other palette, based on rules like the above, OR come up with a programmatic color tuning approach that enables the boosting or tempering of colors
• find a way to address the color adjacency issue, as round robin in a colorwheel order reduces contrast of adjacent slices or bars
• as Caroline says, we should improve on accessibility in eg. charts, to let demand off color coding
• periodically revisit and benchmark the color palette itself (which Design has been doing consistently already); for example, tradeoffs around a fairly homogeneous level of lightness (value) that places more work on chroma, as we seem to do now, vs. relying more on lightness dynamism for accessibility but then we end up with a palette like Tableau's that has rather light and somewhat dark colors alike, which has a certain appeal

[monfera]

Closing it for now, as freedom for further, significant color improvements hinges on more complementary accessibility features