[REVIEW] [Accessibility] Gatherplots review

[domoritz]

Perceivable

• Contrast of all text and elements is sufficient. Geometries and large text must have >3:1 contrast against background. Regular text must have >4.5:1.
• Content has non-visual alternatives. All figures have alt text and videos have transcripts. Equations are screen reader accessible.
• Font sizes for all texts are sufficiently large. Text must not be smaller than 9pt/12px in size. Ideally only minor text is rendered at 9pt (e.g., axis labels) while all other text is larger.
• The article works even if I could not perceive colors. Color should never be used alone to communicate meaning.
• The article can be navigated comfortably with a screen reader. There are no unnecessary accessibility elements (e.g., axis ticks and labels) that clutter the screen reader experience.

Screen reader

• VoiceOver on macOS
• VoiceOver on iOS
• NVDA on Windows
• JAWS on Windows
• Not tested with a screen reader (explain below why not)

Browser

• Safari or other WebKit-based browser
• Chrome or other Chromium-based browser
• Firefox

Comments on Perceivable

All images should have alt texts that describe the content.

The font sizes of the figures is too small, especially on mobile. You can also consider making the figures responsive and change the layout (e.g. horizontal to vertical arrangement of charts) when the screen width changes.

Right now, a lot of the figures rely on being able to differentiate colors. Try using different shapes in addition to color. For example, in figure 4a, you could use different shapes for survived and not survived.

I noticed that the tooltips for references open when the VO cursor is over a reference and then the cursor goes into the tooltip. It clutters the VO experience so it feels like it should be different. CC @cscheid for Quarto

Operable

• If there are interactive elements in the article, they can be operated with a keyboard.
• If there are interactive elements in the article, there are textual instructions for how to use them.
• When navigating over the article with a keyboard, the focus indicator is always visible.
• Interactions that work with a mouse also work on a touch screen.

Comments on Operable

No interactive elements and focus seem to work as expected.

Understandable

• All charts have a title and a description.
• Changes in animated or interactive elements are easy to follow.
• All axes have clear labels (either explicit or implicitly given the context).
• There are legends for all charts that need them.

Comments on Understandable

The visual hierarchy of axis labels and axis titles could be improved. For example, increase the font size of titles or make them bold. Some charts already use this style.

Robust

• All buttons use the button tag.
• Headings, paragraphs, and figures are tagged correctly.

Comments on Robust

All good.

Assistive

• The data density of all charts is appropriate. Use clustering or other data reduction techniques if elements compete for space but explain the method.
• Navigation and interaction feels intuitive with all interaction methods (mouse, keyboard, touch).
• All visually apparent features and relationships are described.
• Formatting makes values human-readable. Use , or space to separate thousands. Align numbers to the right. Use the smallest, appropriate number of significant digits.

Comments on Assistive

The density of labels is too high in some cases and the marks are very densely packed.

The formatting of some numbers could be improved.

Flexible

• The article is readable on a phone.
• The article can be zoomed and font-sizes change appropriately. The layout of the page does not break when zooming in or changing the font size (e.g., using CMD + or CMD - on macOS).
• Long animations (if they exist) can be paused or stopped. Animations should be videos, not GIFs as they cannot be paused.
• Style and charts use a consistent and familiar design (fonts, colors, etc). Interaction defaults should be consistent for all interactive elements.

Comments on Flexible

As mentioned before, the font sizes in figures are too small on phones.

I found it somewhat difficult to follow some of the charts because they use very different visual styles (fonts, colors, font sizes, etc). This probably happened because the charts use different libraries. It would be good to use one familiar style. This would greatly improve understandability.

ORCID

0000-0002-3110-1053

[cscheid]

thanks for the ping, @domoritz. I'll open a quarto issue to track.

[mjskay]

Thanks for this @domoritz!

I have some questions about some of the criteria and how we can make them work well. Adding @frankelavsky too since I think his opinion would be valuable.

My first thought is: it would be helpful to have a sense of what is critical or not. Perhaps a distinction between "minimum requirements" versus "nice to have". Then we could give a paper a rating like "F (fail) / B (pass) / A / A+" or something like that in an infobox at the top of the paper. When I look at Chartability, I see some issues listed as "critical", maybe that's a starting point for establishing such levels?

Some other thoughts:

• This criterion strikes me as too strict: The article works even if I could not perceive colors. Color should never be used alone to communicate meaning.

  This is a difficult criterion to have in a visualization journal, and when I look at Chartability it is not listed as "critical" and comes with this qualification:

  While this standard is very difficult for the field of data visualization to wrestle with, there is also little research that explores effective strategies.

  It is a little difficult to say "solve this problem in your paper, but we don't know how". I can imagine a number of cases where this criterion will difficult or impossible to be fulfilled (e.g., research on color, complex visualizations of multivariate data where other channels have run out, etc). I agree it is worth striving for, and for the paper at hand is probably easier to achieve than some others, but given the lack of existing good solutions it is a high bar to ask of all authors. Perhaps we can soften the language and/or make it a non-critical criterion (like in Chartability).

• I wonder about this comment:

  The font sizes of the figures is too small, especially on mobile. You can also consider making the figures responsive and change the layout (e.g. horizontal to vertical arrangement of charts) when the screen width changes.

  Totally agree about desktop fonts being too small in several figures. I'm wondering what general advice we can give for folks on mobile --- responsive visualization will be hard or impossible in many cases (e.g., screenshots of desktop VA systems, or people generating charts from static vis tools). For folks using static vis tools, it would be cool if there was a quarto solution where they could provide alternate plots for the same chunk that could be selected responsively (@cscheid?). However, I am wary of placing strict requirements that people use responsive vis tools, since the vast majority of such tools can't do that. So I also wonder if a softer version of this criteria would be considered satisfied if the vis is readable on desktop and at least zoomable on mobile, even if not responsive?

• Lastly, because the description text is stripped when the review is generated, the helpful links to the Chartability sections visible on the review form are not available to authors to get more information. Maybe we could add links to the corresponding information in Chartability to each checkbox item?

Thanks again for all of this!

[cscheid]

For folks using static vis tools, it would be cool if there was a quarto solution where they could provide alternate plots for the same chunk that could be selected responsively (@cscheid?)

We'd love to do this, but it'll be a while until we can get a design going. There's a 1.4 feature that lets you do something almost like what you want, but it would take multiple renders to get the different outputs (specifically, you can now use https://quarto.org/docs/authoring/conditional.html with checks for arbitrary metadata key-value pairs). Do note, critically, that this is not responsive. But I'd love to figure out a solution for you folks.

[nickelm]

I have just pushed changes addressing this accessibility review: #6. Here follows the revision response itself. Please let me know if some other format is expected for this!

All images should have alt texts that describe the content.

I have improved the alt-text for all of the images in the article.

The font sizes of the figures is too small, especially on mobile. You can also consider making the figures responsive and change the layout (e.g. horizontal to vertical arrangement of charts) when the screen width changes.

I have tried to make the images responsive by turning on the fig-responsive setting in the format as well as splitting compound images into component pieces so that they can be better displayed on mobile.

However, I have chosen not to modify images drawn from the Gatherplot implementations. This means that some of the labels, such as in Figure 1, are kept in their original size. Given that an interactive article is primarily intended to be viewed online, it should be possible to see images at full size.

Right now, a lot of the figures rely on being able to differentiate colors. Try using different shapes in addition to color. For example, in figure 4a, you could use different shapes for survived and not survived.

While I appreciate this feedback, this functionality would require modifying the visualization and is thus not part of any of the Gatherplot implementations. I leave such features for future work.

The visual hierarchy of axis labels and axis titles could be improved. For example, increase the font size of titles or make them bold. Some charts already use this style.

As discussed above, I have chosen to not modify the original images from our Gatherplots implementation.

The density of labels is too high in some cases and the marks are very densely packed.

See above; we have chosen not to edit the visualization images but rather to make them larger.

As mentioned before, the font sizes in figures are too small on phones.

I have tried to address font sizes to the extent possible.

I found it somewhat difficult to follow some of the charts because they use very different visual styles (fonts, colors, font sizes, etc). This probably happened because the charts use different libraries. It would be good to use one familiar style. This would greatly improve understandability.

This is a fair point. Unfortunately, we are presenting results from both our old Gatherplots implementation (no longer accessible), Observable Plot, and our own new Gatherplot implementation. They all provide slightly different functionality that I want to demonstrate in the article.

I hope I have addressed all of the concerns, but I am happy to iterate on any issues that still persist.

[domoritz]

Thanks for the updates. These are great and I agree that it doesn't make sense to change the figures for the old implementation.

I found three issues that need to be fixed.

First, the issue with tooltips @cscheid commented on in #10 (comment) still exists and it making it almost impossible to navigate the document with a screen reader. This needs to be fixed in quarto but until it is, I would suggest disabling tooltips for references.

Second, it's great that you broke up the images but the threshold for wrapping should be lower. Can you set a minimum image size? See how the images get super small on a narrow screen.

Lastly, the screen reader experience for the plot charts is too noisy. Try navigating the document and you will find that all tick marks are navigated. Axes should be described in one AX element per axis or skipped and described in the description. Also, each mark in the scatterplot becomes its own accessibility element. It's probably best to group them so that one can skip over the group to reduce noise. See the last comment in the perceivable block at the top: "The article can be navigated comfortably with a screen reader. There are no unnecessary accessibility elements (e.g., axis ticks and labels) that clutter the screen reader experience."

[cscheid]

First, the issue with tooltips @cscheid commented on in #10 (comment) still exists and it making it almost impossible to navigate the document with a screen reader.

Can we have a followup about this at quarto-dev/quarto-cli#6500?

[nickelm]

Thanks for the updates. These are great and I agree that it doesn't make sense to change the figures for the old implementation.
I found three issues that need to be fixed.

First, the issue with tooltips @cscheid commented on in #10 (comment) still exists and it making it almost impossible to navigate the document with a screen reader. This needs to be fixed in quarto but until it is, I would suggest disabling tooltips for references.

I have disabled the tooltips for now. For the record, this is done by adding the following options to the HTML format block:

    citations-hover: false
    footnotes-hover: false
    crossrefs-hover: false

Second, it's great that you broke up the images but the threshold for wrapping should be lower. Can you set a minimum image size? See how the images get super small on a narrow screen.

There seems to be no easy way to do this and maintain responsiveness. For now, I have added a min-width in a CSS class (.accessible-fig), but the responsive behavior no longer works. Pointers on how to achieve this are welcome, but at least the new version is easier to read.

The results in Figure 12 and 13 are still in three-column format, but if need be I can change this as well.

Lastly, the screen reader experience for the plot charts is too noisy. Try navigating the document and you will find that all tick marks are navigated. Axes should be described in one AX element per axis or skipped and described in the description. Also, each mark in the scatterplot becomes its own accessibility element. It's probably best to group them so that one can skip over the group to reduce noise. See the last comment in the perceivable block at the top: "The article can be navigated comfortably with a screen reader. There are no unnecessary accessibility elements (e.g., axis ticks and labels) that clutter the screen reader experience."

Understood. This is an Observable Plot chart, however. I don't know how to change this short of reimplementing the scatterplot myself.

[domoritz]

There seems to be no easy way to do this and maintain responsiveness. For now, I have added a min-width in a CSS class (.accessible-fig), but the responsive behavior no longer works. Pointers on how to achieve this are welcome, but at least the new version is easier to read.

I would suggest using css grid and change the number of columns based on the page width (using a media query). Alternatively, you could use fixed width images so that the number of columns changes based on the width as well.

Understood. This is an Observable Plot chart, however. I don't know how to change this short of reimplementing the scatterplot myself.

Then I would say that we better hide the ax elements here and just rely on alt text until observable plot improves.

[domoritz]

I sent a pull request to improve plot to generate better aria values for axes by default: observablehq/plot#2018. You should still add custom aria for marks since plot doesn't do that automatically yet.

[mjskay]

Is there a simple solution here that doesn't require hacks around what quarto / observable plot are doing? I'd rather not require authors to implement a bunch of hacks around the systems they are using, as that adds a lot of additional burden on the author side. It seems like wasted effort to spend a lot of time on hacks around accessibility issues that are likely to eventually be fixed upstream --- since once they are fixed, we can just recompile the paper here.

To me this suggests not dropping down to custom CSS / HTML / etc to fix this, but to ask what are reasonable short term fixes within the bounds of quarto / observable in expectation that long term this will be solved upstream in observable plot and/or quarto (which both seem receptive to such fixes?).

[mjskay]

(FWIW, Dom, I really appreciate the work you're doing to push us to be better --- I just want to balance that against author burden in expectation of upstream changes. Since we're not a traditional there-is-exactly-one-canonical-version-of-a-paper journal, getting it right here isn't a one-shot deal, but a process.)

[cscheid]

@mjskay It's also possible that whatever short term fixes we adopt over time can be incorporated to the Quarto template. (See the tooltip disabling above for a concrete case)

[mjskay]

@cscheid definitely! We can collect those changes here: journalovi/jovi-template-quarto#12

[domoritz]

Absolutely. I think my fix that landed way faster than I expected is already good enough. As long we keep updating observable plot I think what we have now is good enough.

Hopefully I can make time to look into automatic aria descriptions for marks (similar to what we already have out of the box in Vega-Lite) for Plot at some point as well.

I think that's what I love about living documents in e.g. jovi. We can make these improvements at the library levels and all documents that use the libraries can be updates to use the improvements.

[domoritz]

I think that concludes the review for this paper. I'll close the issue. Thanks everyone for the inputs, suggestions, comments, and making changes!

[mjskay]

awesome, thanks so much @domoritz!