Bringing user "actions" to the web (similar to VoiceOver and Switch Control custom actions)

[juandopazo]

Hi all,

I've been working on a certain productivity web app that behaves very much like a native application. When working on making this web app easy to use with screen readers I learned of an iOS VoiceOver feature that is really useful: actions. For anyone not familiar with VoiceOver (I might send this issue to people with varied levels of involvement), here's how this feature works:

• Left and right swipes navigate across focuseable elements.
• iOS has a rotor UI that lets the user switch between different modes of navigation. For example, the user can chose headings. When in headings mode, swiping up and down focuses the next and previous headings in the document.
• When an element has actions associated, VoiceOver indicates it by saying "actions available".
• When switching the rotor to actions, swiping up and down rotates over a list of available actions to be taken on that element. Double tapping activates the action.

An example of this could be a list of conversations in a chat application, where the actions can be "mute", "archive" and "delete".

In a way, it's like a context menu for screen readers. So maybe this could be built on top of the HTML5 context menu? Other than that, it's very likely that the API would have to be similar to the context menu, with invisible elements related to another element.

If there's interest in this, I can help with any spec work and reaching out to browser engineers. The Accessibility team @ Yahoo will probably be involved and they have expressed interest in helping make this happen.

[jnurthen]

(Chair hat off) I have often wanted this myself. I would enthusiastically support such a proposal. (Chair hat on) This doesn’t really fit the scope of 1.2 which is limited to role parity but could certainly be a 1.3 feature.

[mcking65]

I agree with the jist of what you are saying, but am pretty sure that we don't need any significant new ARIA features to solve it.

So, to test my hypothesis, my first question is how you are providing access to the same context actions for keyboard users when there is no assistive technology running?

[juandopazo]

That's a good question. The more I think about it, the more this looks like a context menu. If the context menu HTML elements were available, I'm sure Safari/iOS could implement actions on top of it without any changes. However, I just realized the HTML context menu spec was dropped. They seem to be favoring using JavaScript to replace the browser's context menu. This option seems out.

The key feature here is allowing assistive technologies to operate on certain elements with minimal navigation. Keyboard users would probably be able to do the same operations with simple shortcuts. However, touchscreen devices seem at a disadvantage here.

[juandopazo]

And also, what makes this feature relatively unique is that, in terms of the DOM, the elements that would activate an action should not remove focus from the active element.

[jamesalley]

I think it's important to note that what Juan is proposing already exists on the native app side. Yes, we can make accessible web apps without this (we always do our best with what's available), but they won't be quite as easy to use (or develop) and they won't have parity with native apps. I think this proposal, or whatever it leads to, is important so we can build common interaction paradigms whether they are native or web.

[mcking65]

The reason I don't think we need much in the way of new ARIA to support this is that I believe it's feasible with a combination of a token for aria-haspopup and the AOM user action event called accessiblecontextmenu.

The AOM event would let the web page listen for a call for a context menu that comes only from assistive technologies. We could let assistive technologies know that a custom menu for assistive technology users is available with a new token for the aria-haspopup property, e.g., "atmenu". While this would not yield an experience that is identical to VoiceOver, it would be equivalent in function.

It would be feasible to make an identical experience with the addition of some other events to AOM. For instance, their could be events for show next context action and show previous context action. Then, when a VoiceOver user moves the rotor to actions and swipes down, VoiceOver would send the web page the show next context action event. When swipe up is detected, VoiceOver would send the previous context action event. I don't know if it would be necessary to have a separate activation event for the context action or if the current accessibleclick would be sufficient. In this case, the web page could reveal the presence of the actions to the screen reader with something like an aria-haspopup token value of "atactions".

[jnurthen]

(note the current AOM proposal is simply to use the contextmenu event for this)
Marking this as 1.3 as it doesn't fit into 1.2 scope and really requires AOM to be implemented to resolve.

[cookiecrook]

Changed the title to "user actions" rather than "VoiceOver actions."

1. Even in the Apple ecosystem, custom accessible actions aren't limited to VoiceOver. For example, Switch Control and Assistive Touch users can trigger these.

2. I'm reasonably certain other non-Apple ATs have similar behaviors conceptually at least, though the implementation are not similar.

3. This may be useful outside the subset of accessibility-specific actions. If ARIA WG proposes something, there should be some discussion with TAG (@alice for example) and other HTML/DOM stakeholders outside the accessibility working groups.

[cookiecrook]

Privacy note: Some of this work has been discussed in AOM, but it's not immediately clear how to do so without "outing" that the user has assistive technology installed and active.

[cookiecrook]

Related 10-minute WWDC video explaining custom actions: https://developer.apple.com/videos/play/wwdc2019/250/

[cookiecrook]

And if you have time to watch a second 10-minute video:
https://developer.apple.com/videos/play/wwdc2020/10116/

[cookiecrook]

RE: AOM, common UI controls may be implemented by using de facto keyboard shortcuts a la AOM #166: left/right arrows to expand and collapse tree items.

[cookiecrook]

In discussions this week, @chrisosaurus mentioned that @robdodson had posted about Command Palettes for the Web, which should be considered in this context.

[cookiecrook]

Some other ideas discussed in various contexts:

• new JavaScript custom event listener with a localized label... maybe something like target.addCustomEventListener(type, listener, label [, useCapture]); Note the new label parameter, which could be used by assistive technology like the actions available to VoiceOver, Switch Control, etc. Mainstream UI (perhaps a context menu) should exist too, so that triggering the action does not "out" a user of AT.
• @alice suggested a way to link an element to other elements that perform the related action specifically for that element. For example, if there is a hover visible "reply" button in list view of an message inbox, there's an association between the message and the already-labeled "reply" button for that message. I suggested this might be done with an new ARIA attribute with an IDREFS value type... Perhaps <div id="msg23" aria-actionelements="reply23 delete23 markspam23">... AT could resolve the association and trigger a default 'click' action on the related action elements... I think that web developers would need to ensure these related elements are clickable at any time, not just on mouseover, for example.

[jnurthen]

Proposal for Part 2 of discussion on Jan 7, 2021?
Please let me know if this doesn't work for interested folks

[jnurthen]

we will not be doing part 2 of this tomorrow. We will discuss when it will take place in the standard ARIA call tomorrow.

[jnurthen]

Part 2 of this will be tomorrow Jan 14, 2021.

[cookiecrook]

[placeholder] after the ARIA calls fill out Matt's idea re: rendering a generated style-able pseudoelements associated with the custom element listener idea. e.g. don't render actions in the browser chrome, but have some easy default rendering of actions that is 1) obviously in the web page, and 2) easily re-style-able by authors...

[cookiecrook]

Will likely be superseded by

• Secondary actions on items in composite widget roles #1440

[jnurthen]

lets close this and follow in #1440