WCAG Accessibility

[pdodde]

I'm looking for a solution to add a WCAG tool to our CI pipeline and not having a lot of luck. This project seems like a good place for it but I don't see any mention of accessibility. Have there been any discussions or work done or would there be any interest in adding that to this tool?

[paulirish]

@pdodde we are currently running axe-core 1.x on the page and reporting 5 audits from it.

see the following:

• gatherers/accessibility.js
• audits/aria-allowed-attr.js
• audits/aria-valid-attr.js
• audits/color-contrast.js
• audits/image-alt.js
• audits/label.js
• audits/tabindex.js

We are interested in upgrading to axe-core 2.x and especially 3.x (see dequelabs/axe-core#240).

And while we report only these 6 axe violations now, it wouldn't be hard to add a few more audits to report on the other violations that axe-core reports. We'd love your help with that, if you're interested.

[ebidel]

CC @robdodson

[pdodde]

Thanks. That is very helpful. I will start digging in and see where we get.

[brendankenny]

for the CI side of things, we have an tool in the project called 'smokehouse' that will run lighthouse and return a non-zero exit code if the results of the run don't match an expectations file. One example of a script that runs it is here running against this expectations file.

Another option is just to do the testing yourself. @justinribeiro has an example of how to test lighthouse audit results with mocha.

[robdodson]

@pdodde

Keep in mind, a tool like axe-core can help with only certain WCAG issues. For example, it can easily tell if a button is missing a label, or if something has bad color contrast. But it cannot tell if a custom control is inaccessible. For example, if you build your own custom checkbox out of a div, <div class="check">, it won't be able to infer that you were trying to build an analog to the native checkbox. A custom checkbox has a lot of keyboard behavior and ARIA it needs to support. There really isn't any tool that can catch if this support is missing in a custom control, so it means you'll always need to do some level of manual QA with a screen reader.

[wardpeet]

Perhaps it's fine to add a disclaimer to the report saying that this doesn't cover all the things and a manual run is necessary as well like @robdodson says.

[paulirish]

@pdodde btw some more recent work in this area.

I would take a look at the work @robdodson has done: https://github.com/GoogleChrome/lighthouse/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Arobdodson

Would be very interested in your thoughts and contributions if you think we could do something better.

thanks!

[pdodde]

Thanks. I've been sidetracked but hope to get back to this soon.

[jasonday]

@robdodson - status of adding additional checks? Is there an inventory of what axe rules are included and identification of rules slated to be included, versus not?

Due to this limited subset of axe rules, the scoring is doubly arbitrary. Scoring is somewhat arbitrary based on the need to do manual checks in conjunction, but with only a limited ruleset, the scoring becomes misleading because there's no indication that the ruleset is limited.

[robdodson]

@jasonday we have 35 tests now, they all live here: https://github.com/GoogleChrome/lighthouse/tree/master/lighthouse-core/audits/accessibility

The test files names correspond to this table of axe rules:
https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md

Basically we're running aXe with all of its wcag2a and wcag2aa checks plus a few best practices and a handful of esoteric tests turned off.

lighthouse/lighthouse-core/gather/gatherers/accessibility.js

Lines 23 to 35 in c780dfa

	values: [
	'wcag2a',
	'wcag2aa',
	],
	},
	rules: {
	'tabindex': {enabled: true},
	'table-fake-caption': {enabled: true},
	'td-has-header': {enabled: true},
	'area-alt': {enabled: false},
	'blink': {enabled: false},
	'server-side-image-map': {enabled: false},
	},

Scoring is somewhat arbitrary based on the need to do manual checks in conjunction

This will always be the case. aXe can only test a small subset of things. You'll always need to do manual usability tests.

but with only a limited ruleset, the scoring becomes misleading because there's no indication that the ruleset is limited.

Testing WCAG2A and WCAG2AA is about as comprehensive as we can get. As aXe adds new rules and support for things like Shadow DOM we'll be rolling those in as well.