iOS Sidecar VTT TextTrack toggling does not work

[shlokamin]

Current behavior

When attempting to toggle captions on and off via the selectedTextTrack prop, the video stops and freezes. I can no longer play the video after attempting to toggle. This happens for all videos I have tried.

Note, toggling captions works beautifully on Android.

Reproduction steps

Currently I have a state variable for captions. I control the toggle through a button and set the type of selectedTextTrack to "disabled" or "title".

i.e.
In the video component:
selectedTextTrack={this.state.captions}
textTracks={[
{
title: "English",
language: "en",
type: TextTrackType.VTT, // "text/vtt"
uri: "https://bitdash-a.akamaihd.net/content/sintel/subtitles/subtitles_en.vtt"
},

In a button press event:
toggleCaptions = () => {
this.setState({
captions:{
type: this.state.captions.type === "disabled" ? "title" : "disabled",
value: "English"
}
})
}

Platform

Which player are you experiencing the problem on:

• iOS

[cobarx]

@ashnfb This might be the same problem you were having with the video not playing.

[cobarx]

Opened #1157 to address this. I can't figure out a way to disable all text tracks atm...right now I can only figure out how to make the player switch between tracks. Hoping to find an AVPlayer expert who can answer this, as soon as I get feedback I'll merge the fix.

In the meantime, the only workaround I can think of is to create an empty VTT file and select that when you want to disable captions.

[nbennink]

Any update on this? I still experience this issue with the newest version of react-native-video.

An empty VTT file works, but it isn't good practice. I don't have the options to upload an empty VTT file to a server either so it will never work for my production app (local uri doesn't work).

The code in RCTvideo.m seems fine, all the text tracks get disabled so I'm not sure what's wrong.

[shlokamin]

@nbennink definitely not ideal but what I've done is created an empty VTT locally and used that. We use rn-fetch-blob to write an empty VTT locally and we just read from that to simulate the empty captions.

Again, not ideal and definitely very hacky, but we couldn't figure out any other solution. If somebody has, please comment and me know!

[nbennink]

@shlokamin I already have rn-fetch-blob in my project and I've created the vtt file in my project. But how can I add the file as uri to the textTracks. Require is not possible for VTT files, and readFile from rn-fetch-blob returns: File does not exist for me.

[shlokamin]

@nbennink you have to prefix the uri with "file://". You shouldn't actually need to read anything from fetchblob- we did something like this:

fetchCaptions = () => {
    if(this.captionsExist()){
      let dirs = RNFetchBlob.fs.dirs
      RNFetchBlob
      .config({
        path : dirs.DocumentDir + '/captions.vtt'
      })
      .fetch('GET', this.captionable.captions_file_url, {
      })
      .then((res) => {
        this.setState({captionsFilePath:`file://${res.path()}`})
      })
    }
  }

fetchEmptyCaptions = () => {
    if(this.captionsExist()){
      let dirs = RNFetchBlob.fs.dirs
      RNFetchBlob
      .config({
        path : dirs.DocumentDir + '/empty_captions.vtt'
      })
      .fetch('GET', this.captionable.captions_file_url, {
      })
      .then((res) => {
        return res.text()
      })
      .then((text) => {
        let path = dirs.DocumentDir + '/empty_captions.vtt'
        let emptyCaptionsFile = getEmptyCaptions(text)
         RNFetchBlob.fs.writeFile(path, emptyCaptionsFile, 'utf8')
          .then(()=> {
            this.setState({
              emptyCaptionFilePath: "file://" + path
            })
          })
      })
    }
  }

and then in the Video component:

<Video>
  textTracks={
                  this.captionsExist() ?
                    [{
                      title: "English",
                      language: "en",
                      type: "text/vtt",
                      uri: this.state.captionsFilePath
                    },
                    {
                      title: "empty",
                      language: "en",
                      type: "text/vtt",
                      uri: this.state.emptyCaptionFilePath
                    }]
                    : null
                }
</Video>

We basically just write text to a local file using fetch blob and the Video component reads from it. Does that makes sense?

Also, for the empty captions, we would get an error from the Video component if the empty VTT was not spaced correctly, or if the time stamps were not identical to what the original VTT had. So basically we had to create a new empty captions file that copied all of the timestamps from the original VTT, but left the text blank. We originally tried just using a VTT for empty captions like this:

WEBVTT

00:00:00.000 --> 01:50:17.910

so that captions would be blank for all videos, but for some reason that didn't work. I would try this again though. It was definitely a pain and very hacky copying all of the timestamps from the original VTT, but that was the only way for us to get it working :/ if you'd like the code for this let me know I can figure out a better way to share it.

I had forgotten all of the crazy stuff we had to do to get this to work, I only just remembered because I went through our code this morning.

If anyone has been able to do this in a better way, please comment!

[ashnfb]

@shlokamin Nice solution Amin! thanks for sharing. I'm going to use that :)

[ashnfb]

@nbennink Sidecar captions are a hack because iOS doesn't provide a formal way to do them. Like @cobarx, I looked but wasn't able to find a clean solution to disabling captions. There's other problems with this hack too - videos with sidecar captions won't airplay either :(

[shlokamin]

@ashnfb thanks! The solution I'm using does not work with airplay? Unfortunately I don't have an apple tv to test on but that's definitely a bummer if that's the case :(

[shlokamin]

@ashnfb @cobarx since there does not seem to be a formal way to disable captions in iOS, do y'all think it makes sense to to take what I've done and implement it as part of the Video component? While it's hacky, there doesn't seem to be a better way, and it seems silly to have consumers write logic that's all going to be the same. If so, I'd be happy to open a PR!

[ashnfb]

@shlokamin you read my mind. I think having it implemented by default makes sense. My worry is adding hacks to the core codebase - it makes maintenance harder in the long term. Let's get @cobarx's opinion?

[nbennink]

Or we add an empty VTT file in the package and load it when you pass type="disabled". Not the best solution either. But I think that it's better to fix the problem in the core (in RN Video) and not within all the apps that want to use this.

I'm not sure how the subtitles work with Airplay. I can test it next week to see what happens.

[ashnfb]

@shlokamin the subtitles don't work with airplay at all because we're using a MixComposition which for whatever reason, airplay doesn't support.

[alanlanglois]

any news?

[nbennink]

I have it as part of the video component in a local branch now. I can make a PR to merge it to master. But it might not be the best option to add this "hack" to production as suggested by @ashnfb. On the other hand: it's better than nothing or hacks in your own project because it's definitely an issue with the Video component itself.

[alanlanglois]

Yep, I'll do it in the project first.
If there is no other way to get ride of this issue, the hack should be included in the lib or at least documented.
People facing this issue, shouldn't dig untill here to find answers to this bug :)
Also another possibility could be to create a lib that maps the react-native-video props and fix the problem having the trick inside.

[shlokamin]

I agree that if we cannot find a more suitable solution that the hack should be documented and included as part of the repo. This is an important feature for accessibility purposes, for the app I am building it is actually a legal requirement. I've spent a lot of time trying to figure out how to get side car text tracks to work natively in iOS but I have not found any luck. Documentation is sparse and I can't find seem to any working examples. @cobarx what are your thoughts? In the mean time I can open up a PR to add some documentation about this so people don't need to struggle through trying to get this to work with the component out of the box. I should have some time tonight/tomorrow morning.

[alanlanglois]

I'm wondering why the freezed captions are removed when entering fullscreen and leaving fullscreen.
I guess it has something to see with re-rendering, maybe this can help find another trick such as force re-render the component?

[VincentCATILLON]

@alanlanglois That's a good point !

[nbennink]

#1468 is how we fixed it. It's similar to the approach with RNFetchBlob but this time I generate the file when the texttracks get added to the player.

[shlokamin]

@nbennink interesting, I like this better than using RNFetchBlob. Nice!

[CHaNGeTe]

Isn't this an option?
https://developer.apple.com/documentation/avfoundation/avplayeritem/1389610-selectmediaoption?language=objc