Computing the Accessible Backing Node, or a similar test-only interface

[cookiecrook]

Historical context:

• AOM Explainer: Use Case 5: full introspection of accessibility tree
• AOM Explainer: What happened to AccessibleNode?

2023 Developments

In the context of the Interop 2023 Accessibility Investigation, we're making progress on cross-browser accessibility testing using TestDriver/WebDriver based interfaces computedrole and computedlabel.

In order to perform more cross-browser accessibility tests, we'll need interfaces beyond role and label. Some of those current testability shortcomings are outlined in web-platform-tests/interop-accessibility#6 and web-platform-tests/interop-accessibility#21

Rather than hanging a growing list of accessibility properties (computedchecked, computedrequired, etc.) off of element, most AOM stakeholders would prefer this be a new test-only interface that can contain the additional properties, probably something akin to the original intention of AOM's proposed AccessibleNode, but with some key differences. In particular, being a read-only interface used for testing, rather than a read/write interface as a general DOM API.

Initial Investigation

Nothing is being formally proposed yet, but I think what this incubation may entail is the following:

1. A new AccessibleNode interface (peer of Node, not Element) that could be specified in either the HTML spec, or perhaps a new deliverable in the ARIA Working Group.
2. Likely as part of that interface, a list of the property values returned along with the new AccessibleNode.
3. A new WebDriver edit defining how to get a known AccessibleNode (similar to get a known element).
4. A new WebDriver interface on element to Get Computed Accessible Node for document and any given DOM Node.
   • Would likely need to return null except in cases of DOMElement or TextNode where there was a backing AccessibleNode...
   • This differs between implementations, but in general, there is no backing accessibility node for hidden elements, comment nodes, or some other rendered DOM Nodes that are just irrelevant in the associated accessibility tree.
   • Likewise, there are some accessibility nodes that have no representation in the DOM. CSS generated pseudo-elements like ::before and ::after are one of several examples.
5. WPT TestDriver wrappers around that new WebDriver interface, so that computedaccessiblenode can be "got" in a WPT interoperability test...

Clarifications

1. I believe everyone who has discussed this (in AOM, ARIA, WPT, and BTT) is in agreement that this should remain a test-only, read-only interface for the currently foreseeable future. Unlike a prior AOM experimental implementation, this is not intended as a web authoring API.
2. The primary client for these changes will be Web Platform Tests (WPT), but some larger JavaScript frameworks or other Accessibility Testing Tools may also benefit (in their testing contexts) from the proposed changes.
3. The intended interface to AccessibleNode would only be a browser's representation of the accessible node... This does not expose platform accessibility API differences, and in fact, never leaves the rendering engine implementation.
4. Initially, it is understood that there will be some implementation differences between the browsers. In particular, a dump of an accessibility tree walker will be highly unlikely to return the exact same tree between the three major implementations. This is expected. Any implementation differences discovered will be used to further discussion of each issue on a case-by-case basis.
5. Because of the known implementation differences, it is NOT a goal of the WICG AOM group to get to an identical accessibility tree as part of this investigation. If it happens, that's fine, but it does not need to happen in order for this interface to be useful. If we can agree on the estimated 80–90% of places where the implementations align, that's 80–90% more interop testing than we could achieve before.

[cookiecrook]

Once we have some new interface defined, we may need to request new error codes in WebDriver when say, computedrole or another getter is called on a DOM element without a backing accessibility element. See related:

• WPT test harness crashes if test receives "unknown error" from WebDriver  web-platform-tests/wpt#39617

[jcsteh]

Further to @cookiecrook's comment regarding elements not in the a11y tree (originally discussed in web-platform-tests/wpt#38941 (review)):

https://github.com/web-platform-tests/wpt/blob/25cd6eb086db5977ac51f7dee7faafe6772dc9d7/wai-aria/role/synonym-roles.html#L16-L17

These tests (and perhaps others) set role=none/presentation and expect that the computed role will be "none". role="none" excludes the element from the a11y tree, so we can't get the computed role from the a11y engine. This raises the question: what are the expectations for computed role for nodes which aren't in the a11y tree?

Gecko currently returns null, but I can change that to anything: "", "none", whatever. However, it seems to me that there is a subtle difference between role="none" (one very specific way of removing an element from the a11y tree) vs the non-existence of an element in the a11y tree for one of several reasons (role="none", hidden, aria-hidden, ...). Could "none" lead people to think that this is specifically about role="none", when in reality, aria-hidden will return "none" as well? Or will we not return anything and use error codes as @cookiecrook suggests?

That said, I don't want to bikeshed too much here. If WebKit and Chromium are happy with "none" for all cases of a11y non-existence, I can just change Gecko's WebDriver implementation to return "none" and move on. Nevertheless, I thought it'd be good to make a clear, documented decision here.

[jcsteh]

Putting non-existent accessible nodes aside, an open question for me is whether this needs to be defined as a DOM interface or whether it should be WebDriver specific. Currently, in Gecko's WebDriver implementation, computedRole is not exposed on DOM Element at all. Rather, it is an internal Gecko interface (not WebIDL) that is only accessible via WebDriver; i.e. you can't access it from the DOM at all.

As an interesting side note, Gecko does also have an AccessibleNode interface (not used for WebDriver) that was implemented years ago based on a very old version of AOM.

[jcsteh]

Another thing to consider is that when something is dynamically added to the DOM (WPT role tests currently do this), it might not immediately be available in the a11y tree. It's unclear what AccessibleNode should do in that case.

[cookiecrook]

Following up from a mention in the ARIA F2F this week, @jcsteh suggested this might be a static copy of the accessibility element at the time of request. All properties could be static strings including WebDriver identifiers for related accessibility nodes like parent and children.

[cookiecrook]

[Update: I'm being purposefully vague about syntax because I don't want us to get hung up on implementation details yet.]

Discussed a bit with some colleagues. I think all this will require [in WebDriver] is:

• a way to get an accessibility element (if it exists) from the equivalent DOM element
• a way to get an accessibility element by its ID (not the same as element ID)

The reconciliation between DOM tree versus AX tree could be handled using those to method and the existing WebDriver interface to get a DOM element by its WebDriver element ID.

Returned accessibility element can include their property bags (aria attributes, etc.) and AX ID refs to axParent, axChildElements, and associated domElement (if there is one)… that should be enough to build a tree walker and perform automated interop testing of ARIA and related specs.

Perhaps an accessibility extension spec to WebDriver, a la the WebAuthN WebDriver extension spec?

[jcsteh]

It's still unclear to me whether we're talking about a test-only interface accessible via DOM or an interface in the context of WebDriver. That is, do we expect to be able to make these queries in JS code running in the context of the page (only in WPT tests) or only from WebDriver code? The current computedRole and computedLabel methods are exposed only in WebDriver, so they don't have to be implemented as interfaces accessible via DOM (though they could be depending on the implementation).

[cookiecrook]

I don't think that's determined yet and either could work, but as I mentioned above, "perhaps an accessibility extension spec to WebDriver" is the best option?

[alice]

I think I can understand @jcsteh's confusion, based on the original issue description:

A new AccessibleNode interface (peer of Node, not Element) that could be specified in either the HTML spec, or perhaps a new deliverable in the ARIA Working Group.

I think that if the primary goal is to define an API which may be used for WPTs, an AccessibleNode interface isn't really going to be realised anywhere (other than perhaps in TestDriver, which afaik isn't standardised but just kind of exists?)

How did the BTT/WPT folks envision this working?

The primary client for these changes will be Web Platform Tests (WPT), but some larger JavaScript frameworks or other Accessibility Testing Tools may also benefit (in their testing contexts) from the proposed changes.

I think it makes a big difference which frameworks/tools we're talking about - other than Selenium, I'm not sure what testing frameworks actually use WebDriver (tbh I don't know much about WebDriver in general, but my understanding is that it is largely "remote" from the web page itself, i.e. if you're using WebDriver-based APIs, you're requesting data from the web page rather than interacting with the DOM directly).

For something like axe-core, I think we would need either a web-exposed DOM API, or some way to make a DOM API available only in restricted contexts (and I have no idea how we would do that).

Also, fwiw:

As an interesting side note, Gecko does also have an AccessibleNode interface (not used for WebDriver) that was implemented years ago based on a very old version of AOM.

So does Blink :)

[cookiecrook]

@alice wrote:

I think I can understand @jcsteh's confusion, based on the original issue description

Yes sorry. That was also prefixed with a bikeshed disclaimer: "Nothing is being formally proposed yet, but I think what this incubation may entail is…" Trying to get to a shared understanding of an MVP without getting hung up on specific syntax or implementation details is tricky.

I updated my prior comment to purposefully avoid specific syntax, but indicate what I think is the minimum change needed to expand WPT accessibility interop testing beyond computedrole and computedlabel.

In WebDriver, which is used for most accessibility WPT tests at the moment:

• a way to get an accessibility element (if it exists) from the equivalent DOM element
• a way to get an accessibility element by its ID (not the same as element ID)

The backing interfaces could remain an implementation detail until they need to be surfaced.

[cookiecrook]

Some WebDriver extensions do spec IDL interfaces in addition to the post method; see Generate Test Report in the Reporting API WebDriver Extension Spec. However, if I understand correctly, the IDL return value could be defined there or elsewhere, and it could be broadly or narrowly defined.

[cookiecrook]

Probably a way in WebDriver to post a notification (press, trigger action by action label, etc) on an accessibility element, too.

[aleventhal]

As I understand the proposal, what this will test in Chromium is about the same as what a developer sees in dev tools, which is useful in general sense. Notably, it won't be testing platform behavior/properties.

Is this the final destination? Or, in the future, will we be looking to test platform mappings, such as what is described in AAM specs. Is this a step on the path toward that, or is it heading in a different direction?

[cookiecrook]

@alice wrote:

How did the BTT/WPT folks envision this working?

Shopping around still, but will hopefully have something concrete enough to discuss by TPAC.

@aleventhal wrote:

As I understand the proposal, what this will test in Chromium is about the same as what a developer sees in dev tools, which is useful in general sense. Notably, it won't be testing platform behavior/properties.

Yes, the current goal of the WPT Interop 2023 Accessibility Investigation is making sure the browser accessibility internals are mostly the same between browsers (as much as possible with only label and role validation)… Automated tests to confirm the accessibility dev tools in WebKit, Chromium, and Gecko are returning the same values before making it to the platform-specific Accessibility API mappings. This is already yielding engine differences and we're filing bugs as the new tests are implemented.

Is this the final destination? Or, in the future, will we be looking to test platform mappings, such as what is described in AAM specs. Is this a step on the path toward that, or is it heading in a different direction?

I don't think there is precedent in WPT to have a single implementation pass with different expectations per platform... For example, a platform AAM test would require that Chrome on Windows had a different result than Chrome on Mac... I'm not opposed to that, but I don't know of anything like that in WPT, so it would require new investigations.

[aleventhal]

Agree with everything.
My only hesitation with testing internals is that to us they could change for some internal implementation reason, causing test failures, but the change is irrelevant to the final platform output.

[benbeaudry]

Just to clarify, this new AccessibleNode interface should only be allowed to be queried from a WebDriver, right? Or will there be a way for a web author to get the AccessibleNode object through JavaScript? I definitely have privacy concerns if web authors are also allowed to query it, at least for Chromium. But if not, I'm onboard with this WebDriver-scoped approach.

[aleventhal]

After talking to Ben, I decided that I like the direction.
Worst case scenario for us is a test gets checked in that we break because of some subtle change in, say, which nodes we expose as ignored, or which we include in tree (for us, these are 2 different things). In that worst case, we can always update the test to be more forgiving, or we can mark the test as failing so that we can pass CQ, while filing an issue on the test in WPT.

Although being able to test platform attributes/APIs would be amazing, it would be a huge effort, and it would be blazing new ground as @cookiecrook said. We need something in between and this is that. It will add value in the push for more compatibility between browsers.

[jcsteh]

The ignored/generic accessibility object is the thing that worries me most about tree interop testing with Gecko. At this stage, for performance and simplicity, Gecko actually doesn't have a concept of ignored objects. They're either in the tree or they aren't. Anything else is either a real object or a generic. Sometimes, generics get forced into the tree for various reasons. This has no impact on assistive technology, but it will impact strict tree interop testing. That said, all of that is somewhat separate from defining the interfaces.

[aleventhal]

Yep. Maybe we need to build in some convenience methods for walking the tree but skipping ignored/unincluded/generic, applied wherever the UA has the concept. We could argue that generic is non-semantic. So, firstSemanticChild, nextSemanticSibling, semanticChildren, etc.

…

On Thu, Aug 24, 2023 at 5:06 PM James Teh ***@***.***> wrote: The ignored/generic accessibility object is the thing that worries me most about tree interop testing with Gecko. At this stage, for performance and simplicity, Gecko actually doesn't have a concept of ignored objects. They're either in the tree or they aren't. Anything else is either a real object or a generic. Sometimes, generics get forced into the tree for various reasons. This has no impact on assistive technology, but it will impact strict tree interop testing. That said, all of that is somewhat separate from defining the interfaces. — Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKQAZVY4J2E7XIQQDF2OK3XW666RANCNFSM6AAAAAAWV7AN4E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[alice]

We could argue that generic is non-semantic.

This seems to echo the concept of accessibility parents nicely.

[aleventhal]

Agree. In general I think ATs really only care about generics when there are interesting properties on them, or to understand line breaking in the document. At the level of testing ARIA compliance, it's probably usually useful to iterate the tree without them, except in a few cases. A case where we might want to look at a generic within a test could be as simple as testing <div> with various ARIA properties, to make sure support/non-support is as expected based on whether the property is global or not.

…

On Thu, Aug 24, 2023 at 6:47 PM Alice ***@***.***> wrote: We could argue that generic is non-semantic. This seems to echo the concept of accessibility parents <https://w3c.github.io/aria/#dfn-accessibility-parent> nicely. — Reply to this email directly, view it on GitHub <#197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKQAZUCCL5M4D7B5W46M4DXW7KX3ANCNFSM6AAAAAAWV7AN4E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cookiecrook]

@benbeaudry wrote:

Just to clarify, this new AccessibleNode interface should only be allowed to be queried from a WebDriver, right? Or will there be a way for a web author to get the AccessibleNode object through JavaScript? I definitely have privacy concerns if web authors are also allowed to query it, at least for Chromium. But if not, I'm onboard with this WebDriver-scoped approach.

I tried to address that in the original description above.

Clarifications

1. I believe everyone who has discussed this (in AOM, ARIA, WPT, and BTT) is in agreement that this should remain a test-only, read-only interface for the currently foreseeable future. Unlike a prior AOM experimental implementation, this is not intended as a web authoring API.

Does that answer your question? If not, please clarify.

[cookiecrook]

My only hesitation with testing internals is that to us they could change for some internal implementation reason, causing test failures, but the change is irrelevant to the final platform output.

Yes, I agree we should not check in, or we should actively remove any WPT test that fails due to irrelevant implementation details.

Maybe we need to build in some convenience methods for walking the tree but skipping ignored/unincluded/generic, applied wherever the UA has the concept.

Convenience methods in AriaUtils.js or another place seem fine. I don't think the DOM/AX tree mismatches should be masked out of the WebDriver interface though... ARIA's "accessibility parent" concept masks these out for good reason, but it's not a test API.

We could reconcile the trees (where necessary) using the respective IDs, but it will be useful to have access to these DOM or AX elements that don't have a direct counterpart in every implementation. For example, AX-ignored elements in the DOM (role="none"), or render-tree-based views in the AX tree (including pseudo elements).

We could use the full trees (including implementation diffs) to determine the differences, and align the implementations where it makes sense, or note where we agree to disagree and leave as an implementation detail.

[cookiecrook]

More on that from the OP:

4. Initially, it is understood that there will be some implementation differences between the browsers. In particular, a dump of an accessibility tree walker will be highly unlikely to return the exact same tree between the three major implementations. This is expected. Any implementation differences discovered will be used to further discussion of each issue on a case-by-case basis.
5. Because of the known implementation differences, it is NOT a goal of the WICG AOM group to get to an identical accessibility tree as part of this investigation. If it happens, that's fine, but it does not need to happen in order for this interface to be useful. If we can agree on the estimated 80–90% of places where the implementations align, that's 80–90% more interop testing than we could achieve before.

[cookiecrook]

WICG has this in the Friday Sept 15th: 14:30 CET time slot for 60 mins.

[cookiecrook]

Related TPAC Breakout Time Scheduled:
https://www.w3.org/events/meetings/7ea3022f-4424-45da-ba98-2953baf7d997/

[cookiecrook]

Compiling some of the feedback received this week to discuss in the Friday WICG timeslot:

Wednesday breakout on the Interop 2023 investigation:

• Minutes for the breakout session, thanks @wareid for scribing
• After the meeting, @OrKoN mentioned a related use case for accessibility trees and selectors in WebDriver... which is most related to @mzgoddard's issue on an Accessibility tree dump API

Thursday discussion with WebDriver

• Minutes for the WebDriver session, thanks @jgraham for scribing
• Not unanimous whether this should be in core spec or an extension spec. Either could work.
• Interesting point from @jgraham that most proposed features ("get axnode from el", "get axnode by id", "synthesize AT event") could be in both classic and bidi, but a few ("intercept outbound notification" and "get full/partial tree") seem like they would be easier to implement as bidi-only.

[OrKoN]

I have opened a PR to add ability to query by a11y attributes to w3c/webdriver-bidi#660