[discussion] Caption & subtitle support

[cobarx]

I've started working on caption & subtitle support for iOS and after looking at a number of examples, have a fairly good idea of the scope of what's needed.

Text track types

Subtitles: text for spoken audio translating the dialogue into the viewer's native language
Captions: text for spoken audio & sound effects like explosions and car horns for hearing impaired people

Text track shipping methods

Embedded: Included as part of the video file, typically using the CEA-609 format
Sidecar: Loaded from a separate file or URL, typically in the WebVTT or SRT format

Example text tracks

Index	Title	Language	Type	Description
0	fre	French 1	VTT	French subtitles
1	esp	Spanish 2	VTT	Spanish subtitles
2	eng	English 3	CEA	English captions for movie audio
3	eng	English 4	VTT	English captions for director commentary

Likewise, you could imagine something similar with 4 audio tracks: Spanish & French dubbed, English, and English director commentary. You could also have multiple video streams: 480p, 720p, and 1080p. It makes sense to have a similar API for handling all track selections.

System settings

Both iOS & Android offer captions preferences so hearing impaired people can enable captions system wide, however Windows does not. We should honor the system setting as much as possible.

Where ever possible, we want to copy the HTML5 video API, however when it comes to tracks, the API is limited and clunky, so we can roll our own approach.

Prop for selecting text track

selectedTextTrack={{
  type: Type,
  value: Value
}}

Type	Description
"system"	(default) Display captions only if the system preference for captions is enabled
"disabled"	Don't display a text track
"default"	Display what the video player considers the default track. If unable to determine which track, display track 0 Not currently implemented
"title"	Display the text track with the title specified as the value, e.g. "French 1"
"language"	Display the text track with the language specified as the value, e.g. "fre"
"index"	Display the text track with the index specified as the value, e.g. 0

If a track matching the specified Type (and Value if appropriate) is unavailable, no text track will be displayed. If multiple tracks match the criteria, the first match will be used.

But why make it so complicated?

In some cases, some tracks will be incompatible with one platform but not another, so they won't be loaded and the indexes won't line up. Also, you may want to generate the menu of tracks dynamically and let the customer select them. Which brings us to another point...

Prop for receiving text track info

onTextTracksLoaded={function}
The function receives a payload that looks like:

[
  {
     index: 0,
     title: "French 1",
     language: "fre",
     type: "WebVTT"
  },
  { ... },
  ...
}

This prop allows you to create dynamic menus for customers to select the track they want.

Prop for loading sidecar text tracks

I need to do more research on what info is included in the sidecar files and what needs to be provided before I can comment on this.

Prop for selecting audio track

I need to more research here as well, I believe we can make it similar to selectedTextTrack and call it selectedAudioTrack.

Implementation order

• Embedded text track support for iOS
• Embedded text track support for ExoPlayer (for Android)
• Sidecar text track support for iOS - not possible
• Sidecar text track support for ExoPlayer (for Android)
• Send text track info in onLoad event for iOS
• Send text track info in onLoad on ExoPlayer (for Android)
• Document embedded text track support
• Document sidecar text track support
• Document text track info in onLoad event

Need help with

I don't have time to implement support for the old Android or the Windows WPF & UWP players. If people want this it would be great to receive help in these areas.

Appreciate feedback from @akshaygore06 @hitaloramon @huanghanzhilian @sungjkim34 @petrbela

Edit: Fix grammar and add note about selecting the audio track. Add a disabled type.

[cobarx]

I've created a PR that implements text track support for iOS & ExoPlayer in #1049. I'd appreciate a code review and testing from anyone that cares to check it out.

[fdiep]

Hey @cobarx sorry for the late reply to this and the feature/captions thread. Ill jump right in to test this and update you soon.

Thanks!

[fdiep]

I've tried a couple of different props and am unable to get captions working with this stream
http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8

Phone: Pixel 2
Android: 8.1.0

My android/app/build.gradle has

dependencies {
    ...
    compile project(':react-native-video-exoplayer')
}

android/settings.gradle has

include ':react-native-video-exoplayer'
project(':react-native-video-exoplayer').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-video/android-exoplayer')

I have tried the following Props on the Video

selectedTextTrack={{
   type: 'system',
}}

selectedTextTrack={{
   type: 'index',
   value: 2,
}}

selectedTextTrack={{
   type: 'language',
   value: 'eng',
}}

Not sure if I am passing the props down incorrectly. Let me know if you are able to get it to work with the provided stream.

[cobarx]

@fdiep Oops, I was testing it by setting the text track after the video was already playing. I had forgotten to test with the text track set at the beginning. In that case, the text track gets set before the video is loaded and the text track info isn't available. I've pushed a fix to set the text track once the video loads. It's working for me now, can you test again.

[fdiep]

Awesome @cobarx it worked! Ill do some more testing now and update

[cobarx]

I've added the ability to sideload subtitles in ExoPlayer in #1063. It turns out AVPlayer doesn't allow you to sideload subtitles, so we're out of luck on iOS, they have to be loaded in the playlist.

I also added info about the text tracks to onLoad, so you can see what's available.

I'd appreciate testing and code review if anyone wants to take a look.

[ashnfb]

@cobarx I have iOS side loading working in this fork based on your feature/sidecar-text-tracks branch that was merged into master 9 days ago.

• Sidecar text track support for iOS --- it's possible!

Have a look at ios/RCTVideo.m. The solution uses AVMutableComposition as documented here, and is integrated directly into an AVAsset (ie. captions are not an overlay which can result in sync issues)

I have not completed the logic for text track selection, but this is proof that it works. I'll finish the logic in the next 2 weeks.

[cobarx]

@ashnfb Wow, that's fantastic! I had no idea! Being able to use sidecar everywhere makes it a lot more useful.

Looks like you are on the right approach. Do you know if there's a way to address the sync issues? Looking forward to the final version once it's ready.

This is very similar to MergingMediaSource on ExoPlayer. We could even add support for merging audio tracks in at some point in the future, though I don't know how useful that is.

[ashnfb]

@cobarx I know, we're very happy here. There currently are not any sync issues that I know of (although I'm a bit suspicious of AVAssetURL and whether it synchronous / blocking, or async).

For the remaining work, am planning to copy most of your logic for Android. I will let you know how I proceed.

[ashnfb]

@cobarx here's the pull-request that works with iOS sidecar loading, and a fix for Android to respect user permissions
#1109

[shlokamin]

Wow, super thanks, @ashnfb !
I was actually looking into ways to implement this myself and was running into a hole. This is awesome.

[cobarx]

@ashnfb Thank you! I'm starting to take a look at it today, dunno if I have time to go through the whole thing. I'll post my notes on the PR.