[Internal] deprecate old accessibility API in favor of the new one

[yury-s]

The new guide describes an industry standard way for testing accessibility features of a page so we are going to sunset accessibility snapshots in favor of that approach. See also master bug for the effort: #14112

[Westbrook]

Is there any way you could be convinced NOT to sunset accessibility snapshots?

My team already leverages Axe in many ways, and there are just some contexts where it's not going to allow us to test the specific interaction with the accessibility tree that we've been able to achieve via snapshots of the accessibility tree. Whether it's over the course of advanced interactions with the page, or testing into contexts that are new or unexpected (like shadow DOM implementations of various interactions), having access to the accessibility tree constructed by the browser has proved a much needed partner to our consumption of the Axe tooling to date.

[yury-s]

Can you elaborate on the use cases where Axe is not enough and you prefer to use the accessibility snapshots?

Also, do you run the tests that use the snapshots only in Chromium or in other browsers too? The rationale for deprecating this API is basically we are not going to evolve it, it varies between browsers and maintenance cost of it in WebKit and Firefox is non trivial.

[Westbrook]

The main difference is that generally the Axe tools give a boolean result of accessibility, which is great but is really only a baseline. The boolean is also much more "expectations" driven than "reality" driven, which means that the changes that you get differences in your test results x-browser are relatively low, even when the various browser construct very different accessibility trees for the same selection of DOM. For this reason, I definitely test in this way across all three currently available browsers.

When working with the current accessibility snapshots API, tests can also ensure more complex construction of the accessibility tree. In that way, a overly simple custom select might look like this:

<label id="combo1-label" class="combo-label">
  Favorite Fruit
</label>
<div class="combo js-select">
  <div aria-controls="listbox1"
       aria-expanded="false"
       aria-haspopup="listbox"
       aria-labelledby="combo1-label"
       id="combo1"
       class="combo-input"
       role="combobox"
       tabindex="0">
  </div>
  <div class="combo-menu"
       role="listbox"
       id="listbox1"
       aria-labelledby="combo1-label"
       tabindex="-1">
       <div role="option" id="combo1-1" class="combo-option option-current" aria-selected="true">Apple</div>
  </div>
</div>

This is a pretty straight copy/paste from https://www.w3.org/WAI/ARIA/apg/example-index/combobox/combobox-select-only.html so if it failed the Axe test, I'd be quite surprised.

However, the Axe test won't be able to tell me the composed name of the [id="combo1"] would currently calculate to "Favorite Fruit", instead of something more correct to the selection of "Apple". Having the full accessibility snapshot allows me to test for name="Favorite Fruit Apple", realize that there are attributes missing in the above DOM, correct them and maintain them over time and future refactors.

This reality persists across any number of custom user interfaces that require application (or component library) fortification of the accessibility tree. What more, this is particularly important for the fact that my projects use a large amount of shadow DOM and ensuring that these complex accessibility trees are getting accurately constructed in that environment is even more important.

[yury-s]

Thanks for providing detailed explanation. We are not removing the API immediately and will wait for more feedback after deprecating it.

[testPK]

I agree with @Westbrook we are also using accessibility snapshot in our tests and I like it. @yury-s how do I get information for specific element? getFullAXTree returns all the element data from a page. Is there any way I can pass locator and get info on specific element?

[bennypowers]

At Red hat, We rely on accessibility tree snapshots to get a quick understanding of our custom elements' presentation to AT in unit tests. It would be a shame to lose this capability.

[clshortfuse]

We are not removing the API immediately and will wait for more feedback after deprecating it.

This is good to hear, though this issue has been closed and the guide has no mention of using the tree. Validating against AXE that a page has no violations is one thing. Manually testing against the AX tree is another one completely.

I have discovered numerous Web Components + ARIA bugs on both Safari and Firefox already.

• https://bugzilla.mozilla.org/show_bug.cgi?id=1826194
• https://bugs.webkit.org/show_bug.cgi?id=254934

These were discovered manually (see this codepen) and I still haven't filed a couple other bugs yet. I would like to use playwright to help ensure what we expect to be in the AX tree is actually there. For example, text content is blank under certain conditions or outputted the wrong string (improper string concatenation, stripping whitespace).

AXE wouldn't find violations in either of these, because there's nothing to check against. Whether it says role=toolbar or role=section, AXE doesn't care. It's useful, yes, but just a "big picture" check, not entirely useful for testing components.

Edit: To better explain why DOM isn't enough, some components can be built like this:

<x-button role=none>
  #shadow-root
  <input type=button role=button>
</x-button>

We want to ensure at <x-button>, the first non-hidden node is a button. This can be expressed by the tree above, an internal value in ElementInternals.role, or <x-button role=button>. Either of those three structures should pass the to-be-built ARIA test.

[clshortfuse]

@yury-s I was able to write my own accessibility snapshot with newCDPSession() and parse it. That's fine for Chromium, but WKSession isn't exposed for us to pass Page.accessibilitySnapshot directly to Webkit.

Also, I'm trying to diagnose a bug where the Webkit snapshot doesn't send back pressed with <div role=button aria-pressed=true>foo</div>. I'm pretty sure it's from this line:

https://github.com/microsoft/playwright/blob/7e310f79af8732d40e41042d437623f632610509/browser_patches/webkit/patches/bootstrap.diff#L4850C60-L4850C80

But if this is deprecated, should I even start the process of building a PR to fix it?

Are all the patches that make accessibility snapshotting possible going away too?

[AutoSponge]

I agree that axe-core offers a strong "industry standard" for many auto-detectable accessibility issues. But I disagree with deprecating this API. Axe devtools uses the CDP to achieve many of their "pro" feature tests. If anything, Playwright should be augmenting axe capabilities with the CDP, not relying solely on the limited answers axe-core can provide. Sure snapshot testing, in general, is uninspired but that doesn't mean it doesn't have a place in the testing hierarchy.

My request to include the a11y selector was also ignored. Something as simple as querying by implicit role seems to not be something Playwright team does and therefore I question the accessibility testing methodology at work here. So, this doesn't come as a surprise but another reason why I personally will prefer Puppeteer for accessibility-related testing.

[lismith2-cisco]

With the 'new' method I can't tell if a button is labelled 'Edit profile' or 'skdjfskdfds', how is this a replacement?

[Westbrook]

Here the axe-core team talks about accessibility concepts that they just can't cover at this time that are already covered by const snapshot = await page.accessibility.snapshot();.

[bennypowers]

I prepared this talk and these slides (arrow / hjkl to advance slides) which hopefully demonstrates why playwright's ax-tree APIs have utility independent of and complimentary to aXe audits. The approach outlined here seems to have resonated with a number of teams. Please consider investing in the accessibility snapshot API, rather than deprecating it.