Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

fabOnReact · 2022-08-05T13:00:52Z

Summary

Expandable and Collapsible are unique in the Android Accessibility API, in that they are not represented as properties on the View or AccessibilityNodeInfo, but are only represented as AccessibilityActions on the AccessibilityNodeInfo. This means that Talkback determines whether or not a node is "expandable" or "collapsible", or potentially even both, by looking at the list of AccessibilityActions attached to the AccessibilityNodeInfo.

When setting the accessibilityState's expandable property, it should correlate to adding an action of either AccessibilityNodeInfoCompat.ACTION_EXPAND or AccessibilityNodeInfoCompat.ACTION_COLLAPSE on the AccessibilityNodeInfo. This work should be done in the ReactAccessibilityDelegate class's

Currently, this feature is being "faked" by appending to the contentDescription in the BaseViewManager class. This should be removed when this feature is implemented properly.

fixes #30841

Changelog

[Android] [Fixed] - using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State

Test Plan

On some components, the state expanded/collapsed is properly announced on focus, on some it is not.
On some components only the expanded/collapsed state is announced, and not other component text.
Upon change, state change is not always announced.
The accessibilityState's "expanded" field does not seem to work on all element types (for example, it has no effect on 's).
using accessibilityActions it is possible to add an action for expand/collapse, but these are treated as custom actions and must have their own label defined, rather than using Androids built in expand/collapse actions, which Talkback has predefined labels for.

https://snack.expo.io/0YOQfXFBi

Tests 15th August 2022:

Paper Tests
Fabric Tests

Tests 6th September 2022:

Announcing state with custom actions on Fabric (FAIL).
The issue is not a regression from this PR, as documented in #34353 (comment). It will be fixed in a separate PR.

analysis-bot · 2022-08-05T13:29:00Z

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	7,637,392	-509
android	hermes	armeabi-v7a	7,049,372	-519
android	hermes	x86	7,939,356	-516
android	hermes	x86_64	7,911,371	-524
android	jsc	arm64-v8a	9,513,631	-64
android	jsc	armeabi-v7a	8,288,904	-90
android	jsc	x86	9,453,169	-82
android	jsc	x86_64	10,044,414	-75

Base commit: a789ead
Branch: main

analysis-bot · 2022-08-05T14:23:04Z

Platform	Engine	Arch	Size (bytes)	Diff
ios	-	universal	n/a	--

Base commit: a789ead
Branch: main

https://snack.expo.dev/@fabrizio.bertoglio/crabby-graham-crackers

fabOnReact · 2022-08-08T07:03:16Z

This issue will be fixed in a separate PR, it is tracked in a separate task as not a regression introduced in this PR, neither documented or reported in the original issue.

Main Branch - onAccessibilityAction not called on Fabric renderer

on Paper renderer, onAccessibilityAction is correctly called. It is not called on Fabric.

2022-08-08.21-08-17.mp4

The callback is triggered here

react-native/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

Line 366 in 28e9c3b

return TOP_ACCESSIBILITY_ACTION_EVENT;

react-native/Libraries/NativeComponent/BaseViewConfig.android.js

Line 108 in 28e9c3b

topAccessibilityAction: {

Move back in the commit history. Understand when the onAccessibilityAction stopped to work on Fabric

fabOnReact · 2022-08-17T03:43:21Z

extended/collapsed is announced on state change on Paper (PASS) (enable audio)

2022-08-17.11-31-13.mp4

announce the visible text on Paper (PASS) (enable audio)

the same result as with fabric see the corresponding example in #34353 (comment)

announcing state with custom actions on Paper (PASS) (enable audio)

2022-08-17.11-56-35.mp4

fabOnReact · 2022-08-17T09:48:15Z

extended/collapsed is announced on state change on Fabric (enable audio) (PASS)

2022-08-17.17-14-37.mp4

announce the visible text on Fabric (enable audio) (Pass)

2022-08-17.17-44-22.mp4

announcing state with custom actions on Fabric (FAIL). The issue is with main branch #34353 (comment), will be fixed in a different PR.

…script callback onAccessibilityAction

blavalla · 2022-08-31T18:50:08Z

ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

@@ -414,6 +414,8 @@ private static void setState(
              context.getString(
                  boolValue ? R.string.state_on_description : R.string.state_off_description));
        }
+      } else if (state.equals("expanded") && value.getType() == ReadableType.Boolean) {
+        info.addAction(value.asBoolean() ? AccessibilityNodeInfoCompat.ACTION_EXPAND : AccessibilityNodeInfoCompat.ACTION_COLLAPSE); 


Maybe I'm misreading this, but isn't this adding an action of "ACTION_EXPAND" when the state is "expanded"?

It should do the opposite, when a state is "expanded" the only available action should be "ACTION_COLLAPSE" and when a state is "collapsed" the only available action should be "ACTION_EXPAND".

@blavalla Thanks a lot. I added the suggested changes. I also added these changes:

updated the AccessibilityExample (videos)

added setTag and getTag to retrieve and update the accessibility_state_expanded in ReactAccessibilityDelegate

react-native/ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java

Lines 268 to 276 in b80bcb3

@Override

@ReactProp(name = ViewProps.ACCESSIBILITY_STATE)

public void setViewState(@NonNull T view, @Nullable ReadableMap accessibilityState) {

if (accessibilityState == null) {

return;

}

if (accessibilityState.hasKey("expanded")) {

view.setTag(R.id.accessibility_state_expanded, accessibilityState.getBoolean("expanded"));

}

react-native/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

Lines 228 to 235 in b80bcb3

if (host.getTag(R.id.accessibility_state_expanded) != null) {

final boolean accessibilityStateExpanded =

(boolean) host.getTag(R.id.accessibility_state_expanded);

info.addAction(

accessibilityStateExpanded

? AccessibilityNodeInfoCompat.ACTION_COLLAPSE

: AccessibilityNodeInfoCompat.ACTION_EXPAND);

}

added the logic in performAccessibilityAction to announce the change in expanded collapsed state when using accessibility menu without implementation of onAccessibilityAction.

react-native/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

Lines 365 to 371 in b80bcb3

public boolean performAccessibilityAction(View host, int action, Bundle args) {

if (action == AccessibilityNodeInfoCompat.ACTION_COLLAPSE) {

host.setTag(R.id.accessibility_state_expanded, false);

}

if (action == AccessibilityNodeInfoCompat.ACTION_EXPAND) {

host.setTag(R.id.accessibility_state_expanded, true);

}

updated the summary with the new video test cases

further investigated the issue with onAccessibilityAction not triggered on Fabric (summary available here).

It should do the opposite, when a state is "expanded" the only available action should be "ACTION_COLLAPSE" and when a state is "collapsed" the only available action should be "ACTION_EXPAND". facebook#34353 (comment)

allows user to change expand/collapse using the default action available in the accessibility menu

fabOnReact · 2022-09-06T02:51:43Z

Button which keeps control of extended/collapsed state in JavaScript with onAccessibilityAction, accessibilityActions and accessibiltyState (Paper)

2022-09-06.10-40-19.mp4

fabOnReact · 2022-09-06T03:23:29Z

TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Paper)

2022-09-06.11-19-01.mp4

Just for note, the initial available action is collapse. The video was recorded after interacting with the component.

fabOnReact · 2022-09-06T03:41:52Z

TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Fabric)

2022-09-06.11-39-11.mp4

fabOnReact · 2022-09-06T03:49:42Z

TouchableOpacity announces visible text and triggers expanded/collapsed with onPress and accessiblity menu (Fabric)

2022-09-06.11-43-47.mp4

blavalla

This looks great! Thanks for being so thorough here.

facebook-github-bot · 2022-09-28T17:47:55Z

@blavalla has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

react-native-bot · 2022-11-08T20:35:05Z

This pull request was successfully merged by @fabriziobertoglio1987 in 082a033.

^{When will my fix make it into a release? | Upcoming Releases}

…/Collapsible State (facebook#34353) Summary: >Expandable and Collapsible are unique in the Android Accessibility API, in that they are not represented as properties on the View or AccessibilityNodeInfo, but are only represented as AccessibilityActions on the AccessibilityNodeInfo. This means that Talkback determines whether or not a node is "expandable" or "collapsible", or potentially even both, by looking at the list of AccessibilityActions attached to the AccessibilityNodeInfo. >When setting the accessibilityState's expandable property, it should correlate to adding an action of either AccessibilityNodeInfoCompat.ACTION_EXPAND or AccessibilityNodeInfoCompat.ACTION_COLLAPSE on the AccessibilityNodeInfo. This work should be done in the ReactAccessibilityDelegate class's >Currently, this feature is being "faked" by appending to the contentDescription in the BaseViewManager class. This should be removed when this feature is implemented properly. fixes facebook#30841 ## Changelog [Android] [Fixed] - using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State Pull Request resolved: facebook#34353 Test Plan: - On some components, the state expanded/collapsed is properly announced on focus, on some it is not. - On some components only the expanded/collapsed state is announced, and not other component text. - Upon change, state change is not always announced. - The accessibilityState's "expanded" field does not seem to work on all element types (for example, it has no effect on 's). - using accessibilityActions it is possible to add an action for expand/collapse, but these are treated as custom actions and must have their own label defined, rather than using Androids built in expand/collapse actions, which Talkback has predefined labels for. https://snack.expo.io/0YOQfXFBi Tests 15th August 2022: - Paper [Tests](facebook#34353 (comment)) - Fabric [Tests](facebook#34353 (comment)) Tests 6th September 2022: - [Button which keeps control of extended/collapsed state in JavaScript with onAccessibilityAction, accessibilityActions and accessibiltyState (Paper)](facebook#34353 (comment)) - [TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Paper)](facebook#34353 (comment)) - [TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Fabric)](facebook#34353 (comment)) - [TouchableOpacity announces visible text and triggers expanded/collapsed with onPress and accessiblity menu (Fabric)](facebook#34353 (comment)) Announcing state with custom actions on Fabric (FAIL). The issue is not a regression from this PR, as documented in facebook#34353 (comment). It will be fixed in a separate PR. Reviewed By: NickGerleman Differential Revision: D39893863 Pulled By: blavalla fbshipit-source-id: f6af78b1839ba7d97eca052bd258faae00cbd27b

draft solution with example

d1b97ec

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 5, 2022

react-native-bot added Bug Platform: Android Android applications. labels Aug 5, 2022

adding examples from snack

1b2a898

https://snack.expo.dev/@fabrizio.bertoglio/crabby-graham-crackers

adding action expand and collapse

d58a2db

fabOnReact mentioned this pull request Aug 12, 2022

1/2 TextInput accessibilityErrorMessage (Talkback, Android) #33468

Closed

This comment was marked as off-topic.

Sign in to view

fabOnReact added 3 commits August 15, 2022 22:31

Merge branch 'main' into expanded-state

4e5773c

remove useless if statement

01b0f6c

remove example

9fe1630

fabOnReact marked this pull request as ready for review August 15, 2022 15:14

fabOnReact marked this pull request as draft August 15, 2022 15:14

fabOnReact changed the title ~~Android: Expandable/Collapsible State #30841~~ Android: Expandable/Collapsible State Aug 16, 2022

fabOnReact added 3 commits August 16, 2022 19:36

Merge branch 'main' into expanded-state

0839607

Merge branch 'main' into expanded-state

14509d9

Merge branch 'main' into expanded-state

e5b6b07

remove default expand/collapse action, the state is managed with java…

28e9c3b

…script callback onAccessibilityAction

fabOnReact marked this pull request as ready for review August 17, 2022 11:39

facebook-github-bot added the Shared with Meta Applied via automation to indicate that an Issue or Pull Request has been shared with the team. label Aug 17, 2022

Merge branch 'main' into expanded-state

2e707f3

fabOnReact changed the title ~~Android: Expandable/Collapsible State~~ Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State Aug 17, 2022

Merge branch 'main' into expanded-state

2b18135

fabOnReact mentioned this pull request Aug 26, 2022

Android: Expandable/Collapsible State #30841

Closed

blavalla suggested changes Aug 31, 2022

View reviewed changes

fabOnReact added 5 commits September 1, 2022 13:54

Merge branch 'main' into expanded-state

7033471

invert ACTION_EXPAND and ACTION_EXPAND (codereview)

f9b710d

It should do the opposite, when a state is "expanded" the only available action should be "ACTION_COLLAPSE" and when a state is "collapsed" the only available action should be "ACTION_EXPAND". facebook#34353 (comment)

commiting example

8e7168f

adding return type to Component

6f4184f

improving example

266f13b

fabOnReact marked this pull request as draft September 1, 2022 08:15

fabOnReact added 7 commits September 1, 2022 21:09

change Expand/Collapse with standard action

ef30488

allows user to change expand/collapse using the default action available in the accessibility menu

adding expand/collapse custom actions

1160c32

removing expand/collapse actions

9a60946

Merge branch 'main' into expanded-state

125cb45

paper - expand/collapse using onAccessibilityAction

ba171fc

minor change

c6d19e9

adding collapse/expand in performAccessibilityAction callback

b80bcb3

fabOnReact marked this pull request as ready for review September 6, 2022 07:07

update example

914c7a5

blavalla approved these changes Sep 28, 2022

View reviewed changes

facebook-github-bot closed this in 082a033 Nov 8, 2022

react-native-bot added the Merged This PR has been merged. label Nov 8, 2022

theodur mentioned this pull request Jan 4, 2023

BUG - sev-3 - Android - Spin button change of state isn't automatically announced by TalkBack department-of-veterans-affairs/va-mobile-app#3961

Closed

4 tasks

fabOnReact added the Accessibility label Jan 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

fabOnReact commented Aug 5, 2022 •

edited

Loading

analysis-bot commented Aug 5, 2022 •

edited

Loading

analysis-bot commented Aug 5, 2022 •

edited

Loading

fabOnReact commented Aug 8, 2022 •

edited

Loading

This comment was marked as off-topic.

fabOnReact commented Aug 17, 2022 •

edited

Loading

fabOnReact commented Aug 17, 2022 •

edited

Loading

blavalla Aug 31, 2022

fabOnReact Sep 6, 2022 •

edited

Loading

fabOnReact commented Sep 6, 2022 •

edited

Loading

fabOnReact commented Sep 6, 2022 •

edited

Loading

fabOnReact commented Sep 6, 2022

fabOnReact commented Sep 6, 2022

blavalla left a comment

facebook-github-bot commented Sep 28, 2022

react-native-bot commented Nov 8, 2022

	@Override
	@ReactProp(name = ViewProps.ACCESSIBILITY_STATE)
	public void setViewState(@NonNull T view, @Nullable ReadableMap accessibilityState) {
	if (accessibilityState == null) {
	return;
	}
	if (accessibilityState.hasKey("expanded")) {
	view.setTag(R.id.accessibility_state_expanded, accessibilityState.getBoolean("expanded"));
	}

	if (host.getTag(R.id.accessibility_state_expanded) != null) {
	final boolean accessibilityStateExpanded =
	(boolean) host.getTag(R.id.accessibility_state_expanded);
	info.addAction(
	accessibilityStateExpanded
	? AccessibilityNodeInfoCompat.ACTION_COLLAPSE
	: AccessibilityNodeInfoCompat.ACTION_EXPAND);
	}

	public boolean performAccessibilityAction(View host, int action, Bundle args) {
	if (action == AccessibilityNodeInfoCompat.ACTION_COLLAPSE) {
	host.setTag(R.id.accessibility_state_expanded, false);
	}
	if (action == AccessibilityNodeInfoCompat.ACTION_EXPAND) {
	host.setTag(R.id.accessibility_state_expanded, true);
	}

Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

Conversation

fabOnReact commented Aug 5, 2022 • edited Loading

Summary

Changelog

Test Plan

analysis-bot commented Aug 5, 2022 • edited Loading

analysis-bot commented Aug 5, 2022 • edited Loading

fabOnReact commented Aug 8, 2022 • edited Loading

This comment was marked as off-topic.

fabOnReact commented Aug 17, 2022 • edited Loading

fabOnReact commented Aug 17, 2022 • edited Loading

blavalla Aug 31, 2022

Choose a reason for hiding this comment

fabOnReact Sep 6, 2022 • edited Loading

Choose a reason for hiding this comment

fabOnReact commented Sep 6, 2022 • edited Loading

fabOnReact commented Sep 6, 2022 • edited Loading

fabOnReact commented Sep 6, 2022

fabOnReact commented Sep 6, 2022

blavalla left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Sep 28, 2022

react-native-bot commented Nov 8, 2022

fabOnReact commented Aug 5, 2022 •

edited

Loading

analysis-bot commented Aug 5, 2022 •

edited

Loading

analysis-bot commented Aug 5, 2022 •

edited

Loading

fabOnReact commented Aug 8, 2022 •

edited

Loading

fabOnReact commented Aug 17, 2022 •

edited

Loading

fabOnReact commented Aug 17, 2022 •

edited

Loading

fabOnReact Sep 6, 2022 •

edited

Loading

fabOnReact commented Sep 6, 2022 •

edited

Loading

fabOnReact commented Sep 6, 2022 •

edited

Loading