Adapt iOS16+ dictation

[hellohublot]

Summary:

#19687
https://developer.apple.com/forums/thread/711413

When system version is lower than iOS 16, it does not support dictation and keyboard working at the same time, so if we modify the text, the system will immediately interrupt the dictation, so we need to prohibit modification of the text during recording and recognition

When system version is higher than iOS 16, dictation and keyboard can work at the same time, so textInputMode.primaryLanguage is no longer changed to dictation, so we can modify the text during recording, because the system will not interrupt, but we cannot modify the text during recognition, Because the system will temporarily add a _UITextPlaceholderAttachment to display the recognition UIActivityIndicator

Changelog:

[IOS][FIXED] - Adapt iOS16+ dictation judge condition

Test Plan:

Test Code

constructor(props) {
  super(props)
  this.state = {
    value: '',
    logList: [],
  }
}

render() {
  return (
    <View style={{ flex: 1, padding: 50, backgroundColor: 'white'}}>
      <TextInput 
        style={{ marginTop: 20, height: 50, borderWidth: 1, borderColor: 'black' }} 
        placeholder={'please input'}
        placeholderTextColor={'gray'}
        value={this.state.value}
        onChangeText={value => {
          let logList = this.state.logList
          logList.push(value.length <= 0 ? 'null' : value.replace(/\uFFFC/g, '_uFFFC'))
          this.setState({ value, logList })
        }}
        onEndEditing={() => this.setState({ value: '', logList: [] })}
      />
      <FlatList
        style={{ marginTop: 20, maxHeight: 300, borderWidth: 1, borderColor: 'black' }}
        data={this.state.logList}
        renderItem={({item}) => (
            <Text>{item}</Text>
        )}
      />
    </View>
  )
}

Case A: Required < iOS16

1. ensure that iOS Dictation TextInput ends abruptly between words #18890 can work well and dictation will not be interrupted immediately

_case_a.mp4

Case B: Required >= iOS16

1. ensure that iOS Dictation TextInput ends abruptly between words #18890 can work well and dictation will not be interrupted immediately

_case_b.mp4

Case C: Required >= iOS16

1. start dictation
2. then do not speak any words
3. then end dictation
4. verify that onChangeText will callback "\uFFFC" once
5. and then verify onChangeText callback an empty string "" once

_case_c.MP4

Case D: Required >= iOS16

1. start dictation
2. input some text while speaking some words
3. then end dictation
4. and verify that the onChangeText callback work fine.

_case_d.MP4

Case E: Required >= iOS16

1. start dictation
2. say a word
3. and then switch the keyboard to other language
4. verify that dictation will not end
5. continue say some word
6. verify the onChangeText callback work fine.

_case_e.MP4

[dmytrorykun]

Hi @hellohublot, thank you for the PR. Unfortunately we can't merge it because it is using private API _UITextPlaceholderAttachment.

• Could you please add more detailed description of the problem? What you are doing, the expected behaviour, the actual behaviour.
• As for the proper fix, firstly I would try to understand why the following happens 👇

the system will delete _UITextPlaceholderAttachment and replace it with the recognized text, but when we pass _UITextPlaceholderAttachment to Javascript, we will force convert _UITextPlaceholderAttachment object to \uFFFC

[dmytrorykun]

Previous comment.

[hellohublot]

@dmytrorykun

Thanks, everything has been changed, could you please review it again

[dmytrorykun]

@hellohublot could you please rebase this PR on the latest master.

[hellohublot]

@dmytrorykun Thanks, done.

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	8,628,612	-103,977
android	hermes	armeabi-v7a	7,940,708	-103,224
android	hermes	x86	9,115,595	-106,348
android	hermes	x86_64	8,970,203	-104,708
android	jsc	arm64-v8a	9,192,772	-105,017
android	jsc	armeabi-v7a	8,382,326	-104,231
android	jsc	x86	9,251,113	-107,414
android	jsc	x86_64	9,509,354	-105,759

Base commit: a310217
Branch: main

[facebook-github-bot]

@dmytrorykun has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[hellohublot]

@dmytrorykun Hello, Is there anything I can help you with? Please feel free to ask

[dmytrorykun]

Hey @hellohublot. typingAttributesBeforeInsertion is still private API. Even though it is not prefixed with an underscore, it is not publicly documented. Developers are not supposed to use private APIs because it may change or stop working any time without warning.
Try using this routine to detect dictation. Does it work for all the cases?

  BOOL isDictationActive = NO;
  for (int characterIndex = 0; characterIndex < oldText.string.length; characterIndex++) {
    if ([oldText.string characterAtIndex:characterIndex] == NSAttachmentCharacter) {
      isDictationActive = YES;
    }
  }

[hellohublot]

@dmytrorykun Thanks for your reply, but I guess that won't work, because we only want to judge whether there is a _UITextPlaceholderAttachment, so when other NSTextAttachment exists, we can't set isDictatingRunning to true, and there are two [NSTextAttachment new] in our code

1. Since it doesn't start with an underscore, I think this can pass the machine review, because the class here is id, so Apple can't judge whether typingAttributesBeforeInsertion is a method written by ourselves

2. NSOriginalFont in our code can pass the machine review, so typingAttributesBeforeInsertion can also pass the machine review

3. I tried the following two methods, but they can't be used to judge
   https://developer.apple.com/documentation/uikit/uitextinput/1614466-insertdictationresultplaceholder
   https://developer.apple.com/documentation/uikit/uitextinput/1614493-frame

4. Apple only opened dictationRecordingDidEnd and dictationRecognitionFailed, but didn't open dictationRecordingDidBegin, so I think typingAttributesBeforeInsertion is the best way

5. typingAttributesBeforeInsertion is an attribute that has been supported since iOS7+, so it is stable enough. If it is replaced by other methods after many version updates, we can continue to modify it to other methods, because we really can't find a public api

[jpdriver]

could we move isDictationRunning to the View(s) and set it to YES in dictationRecordingDidEnd?

it seems like since Apple apparently only wants us to know when dictation ended or failed, that could be the way to go? 🤔

[hellohublot]

@jpdriver Thanks for your suggestion, I mistook dictationRecordingDidEnd for dictationRecognitionDidEnd so I ignored it, I'll try it tomorrow if it works with insertDictationResult or removeDictationResultPlaceholder

[hellohublot]

@dmytrorykun @jpdriver Thanks for your suggestions, I have changed it to dictationRecordingDidEnd, I thought it was dictationRecognitionDidEnd before, because it is too similar to dictationRecognitionFailed

I also modified the PR description, could you please help me to review it again

[hellohublot]

@dmytrorykun Hi, Is there anything I can help you with? Please feel free to ask

[facebook-github-bot]

@dmytrorykun has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@dmytrorykun merged this pull request in e8b4bb0.