Synchronize video and audio positions when seekTo

[tdc-w]

I use two video players and one audio player to play HlsMediaSource.
I'm trying to switch to video player2 while playing video player1, and synchronize the video and audio positions.
AudioPlayer is always playing.

• videoPlayer1（SimpleExoPlayer）
  • playing video file（playWhenReady=true）
• videoPlayer2（SimpleExoPlayer）
  • prepare video file（playWhenReady=false, STATE_READY）
• audioPlayer（SimpleExoPlayer）
  • playing audio file（playWhenReady=true）

In the above state, if calling videoPlayer2.seekTo(videoPlayer1.currentPosition), the playbackState of videoPlayer2 will be STATE_BUFFERING -> STATE_READY.
In this case, it takes time for videoPlayer2 to become STATE_READY, so the currentPositions of videoPlayer2 and audioPlayer are different, so there will be a gap between the video and audio.

for example

• videoPlayer1
  • currentPosition=1000
• videoPlayer2
  • currentPosition 0 -> 1000（STATE_BUFFERING -> STATE_READY takes 500）
• audioPlayer
  • currentPosition=1500

I want to eliminate this gap and synchronize the video and audio positions.
If buffering occurs when seekingTo, I think position synchronization is difficult.
Is there any good way?
Also, is it possible to eliminate buffering when seekingTo?

[krocard]

ExoPlayer does not out of the box (see next comment) support synchronisation between two ExoPlayer Instances.
Your use case should only use one player, playing audio+video. A MergingMediaSource can be used to merge audio, video1 and video2.

You will then be able to switch from video1 to video2 by defining a custom TrackSelection that will allow you to switch from video1 to video2 seamlessly.
I think this could also be done with TrackSelector, manually overriding the selection to switch from video1 to video2.

@tonihei what would you recommend to switch video for this use-case?

[tonihei]

If I understand the requirements correctly the transition doesn't even need to be seamless. So if you know all sources in advance, @krocard's suggestion is probably the best way:

• Create a MergingMediaSource with audio and both video sources.
• Wait until all the tracks are known (EventListener.onTracksChanged)
• Override the video track as/when required:DefaultTrackSelector.ParametersBuilder.setSelectionOverride(videoRendererIndex, trackGroupArray, new SelectionOverride(desiredVideoGroupIndex, 0))

If you want to swap tracks at playback position 1000, I'd also recommend to use PlayerMessage to trigger your change as explained here: https://exoplayer.dev/listening-to-player-events.html#firing-events-at-specified-playback-positions

Having only one player ensures you automatically get A/V sync because there is only one media clock. If you really need multiple players, then synchronizing them is considerably more complicated, but not impossible:

• The audio player has the primary clock.
• All other players need a Renderer that implements MediaClock. You could override a NoSampleRenderer for example to create one, or possibly also subclass MediaCodecVideoRenderer and implement the interface.
• From getPositionUs() you need to return the media position which can then be audioPlayer.getCurrentPosition().

See also #4549 for a similar question. But note that some information in this issue is out of date because we now have options in MergingMediaSource to adjust timestamps, we have SilenceMediaSource to insert silences, and probably other changes that should simplify this setup.

[tdc-w]

@krocard
Thanks.
At first, I used one ExoPlayer to generate MergingMediaSource1(video1, audio) and MergingMediaSource2(video2, audio), and set these in ConcatenatingMediasource and tried it.
In this case, I used seekTo(windowIndex, position), but I couldn't achieve seamless switching due to buffering.
Are you thinking without using ConcatenatingMediasource?

I haven't tried a custom TrackSelection, so I will check it out.

[tdc-w]

@tonihei
Thank you for advices.
I didn't know how to use TrackSelector, so I'd like to try it.
I will also check PlayerMessage.

I actually tried using multiple players, I thought that this method is not correct, so first I will try again to see if seamless switching can be achieved by using MergingMediaSource.

If it is still difficult to achieve, I would like to try the following.

The audio player has the primary clock.
All other players need a Renderer that implements MediaClock. You could override a NoSampleRenderer for example to create one, or possibly also subclass MediaCodecVideoRenderer and implement the interface.
From getPositionUs() you need to return the media position which can then be audioPlayer.getCurrentPosition().

[krocard]

I forgot to say that if you know in advance how long video1 should play before switching to video2, things are much simpler.

You could use 2 Cliping MediaSource to clip the end of video1 after Xms and clip the beginning of video2 of the same Xms.
Concatenate those 2 clipped videos with ConcatenatingMediaSource and then merge the concatenate videos with the audio with MergingMediaSource.

The switch should be seamless and no class need to be implemented. The drawback is that only 1 video switch can occur and it's timestamp must be known before the video start playing.

[tdc-w]

Regarding MediaSource, video deals with videos from different viewpoint.
For example, front-viewpoint video(video1) and back-viewpoint video(video2).
Switching this viewpoint is done by user operation (buttons, etc.).
For example, a usecase where at first watching a front-viewpoint video(video1) and then switch viewpoints after 10 seconds to switch to a back-viewpoint video(video2).
These two videos have the same audio, only the viewpoint is different.

Is Clipping MediaSource effective even in such a case?

[tonihei]

If you know the switch point in advance, then ClippingMediaSource can work seamlessly. Note that this requires to set the enableInitialDiscontinuity parameter in the constructor of ClippingMediaSource to false and to ensure that the clip starts exactly at a keyframe. Otherwise the player needs to buffer briefly to decode samples.
Related: There is also #3163 that tracks the option to dynamically update the clipping if needed.

If your goal is to instantaneously switch in the moment the user presses a button and still get perfectly seamless transitions, then I'm afraid it's more complicated. In order for a seamless switch to work, the player needs to start decoding samples before they are displayed. That means if you want the switch to happen immediately, the player already needs to have the decoded samples available. This in turn means both videos need to be decoded in parallel all the time.

• You could achieve this with two players as described above. Only one of the players has the surface attached and you switch between them by removing the surface from one player and setting it on another.
• A similar behavior can be achieved by adding two video renderers to the player and adding a custom TrackSelector that assigns each video track to a different video renderer (similar to how it's described in Simplify/allow selection of multiple renderers of the same type in DefaultTrackSelector. #6589). And then again only set the surface to one of the renderer (using PlayerMessage sent to the renderer with MSG_SET_SURFACE).

[tdc-w]

My goal is perfect seamless switching.
The current approach is to have two players.
Since processing such as scale change is required (must use setTransform), two textureViews are defined in the layout file.
Each player sets each textureView.
I change the visibility of textureView when switching players.

You could achieve this with two players as described above. Only one of the players has the surface attached and you switch between them by removing the surface from one player and setting it on another.
A similar behavior can be achieved by adding two video renderers to the player and adding a custom TrackSelector that assigns each video track to a different video renderer (similar to how it's described in #6589). And then again only set the surface to one of the renderer (using PlayerMessage sent to the renderer with MSG_SET_SURFACE).

Is it possible to implement the contents explained above in the case of textureView?

[tonihei]

The things discussed above independent of the view/surface you use.

[tdc-w]

@tonihei
Thanks for explanation.

In order for a seamless switch to work, the player needs to start decoding samples before they are displayed. That means if you want the switch to happen immediately, the player already needs to have the decoded samples available. This in turn means both videos need to be decoded in parallel all the time.

I don't understand this.
Does decoding samples means calling player.setMediaSource() and player.prepare()?
Does it mean that seekingTo does not cause buffering if decoding is complete?

Also, I would like to confirm the above methods.
When implementing with pre-decoding by the first method, use two ExoPlayers, and when implementing with TrackSelector by the second method, use only one ExoPlayer.
Is the above understanding correct?

[tonihei]

Does decoding samples means calling player.setMediaSource() and player.prepare()?
Does it mean that seekingTo does not cause buffering if decoding is complete?

Seeking can never be completely seamless because the player always needs to decode the new samples first. This happens in the renderers on a much lower level than setMediaSource or prepare.

When implementing with pre-decoding by the first method, use two ExoPlayers, and when implementing with > TrackSelector by the second method, use only one ExoPlayer.
Is the above understanding correct?

Sounds right. Note we are happy to give hints on how to approach complex implementations, but we can't really provide detailed app support especially when it goes beyond our usually supported features. So it would be great if you can try to work with what we provided so far.

[tdc-w]

Thanks for reply.
I'm grateful for all the advice I received.
I don't know if it can be implemented well, but I'll try it.

[tdc-w]

I created the AudioRendererWithoutClock, MultiTrackRenderersFactory, and MultiTrackSelector described in #6589.
I got an exception when I tried to run with audio only.

val mediaSource = MergingMediaSource(audioMediaSource)
val trackSelector = MultiTrackSelector()
val renderersFactory = MultiTrackRenderersFactory(this)

player = SimpleExoPlayer.Builder(this, renderersFactory)
    .setTrackSelector(trackSelector)
    .build()

player.setMediaSource(mediaSource)
player.prepare()
player.playWhenReady = true

exception

E/ExoPlayerImplInternal: Playback error
      com.google.android.exoplayer2.ExoPlaybackException: Unexpected runtime error
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:586)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: java.lang.NullPointerException
        at com.google.android.exoplayer2.util.Assertions.checkNotNull(Assertions.java:156)
        at com.google.android.exoplayer2.BaseRenderer.readSource(BaseRenderer.java:394)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.readToFlagsOnlyBuffer(MediaCodecRenderer.java:963)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:811)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:947)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:477)
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.os.HandlerThread.run(HandlerThread.java:67)

I'm investigating what's wrong.

[tdc-w]

It was played if I specified multiple audio files.
Next, I will proceed with verification to process multiple video files

[tdc-w]

@tonihei
By implementing VideoRenderers like AudioRenderers, it seems that multiple VideoRenderers could be added.
I don't understand how to switch renderer, can I do it with PlayerMessage?

And then again only set the surface to one of the renderer (using PlayerMessage sent to the renderer with MSG_SET_SURFACE).

Currently, it seems that video1 and video2 are playing, but only video1 is visible.
I want to make video2 visible at any time.

[tdc-w]

Override the video track as/when required:DefaultTrackSelector.ParametersBuilder.setSelectionOverride(videoRendererIndex, trackGroupArray, new SelectionOverride(desiredVideoGroupIndex, 0))

I understand that it may be possible to achieve it by using setSelectionOverride.
I'm also checking #2343.

[tdc-w]

@tonihei
I have a question about how to use PlayerMessage.

mediaSource

mediaSource = MergingMediaSource(video1, video2)

player setup

private var renderersFactory: RenderersFactory = object : DefaultRenderersFactory(this) {
    override fun buildVideoRenderers(
        context: Context,
        extensionRendererMode: Int,
        mediaCodecSelector: MediaCodecSelector,
        enableDecoderFallback: Boolean,
        eventHandler: Handler,
        eventListener: VideoRendererEventListener,
        allowedVideoJoiningTimeMs: Long,
        out: java.util.ArrayList<Renderer>
    ) {
        super.buildVideoRenderers(
            context,
            extensionRendererMode,
            mediaCodecSelector,
            enableDecoderFallback,
            eventHandler,
            eventListener,
            allowedVideoJoiningTimeMs,
            out
        )
        out.add(VideoRendererWithoutClock(context, mediaCodecSelector))
        renderers = out
    }
}

private val trackSelector = MultiTrackSelector()

player = SimpleExoPlayer.Builder(this, renderersFactory)
            .setTrackSelector(trackSelector)
            .build()

I specified videoRenderer from renderers and called createMessage.
In this case, video2 will be displayed.

player.setMediaSource(mediaSource)
player.prepare()
player.playWhenReady = true

player.createMessage(renderers[0])
    .setType(Renderer.MSG_SET_SURFACE)
    .send()

In this case, video1 will be displayed.

player.createMessage(renderers[1])
    .setType(Renderer.MSG_SET_SURFACE)
    .send()

From the above behavior, I thought that it would be possible to switch videos by changing the renderer specification, but in this case nothing is displayed.

// at first
player.createMessage(renderers[0])
    .setType(Renderer.MSG_SET_SURFACE)
    .send()

// when user pressed button
player.createMessage(renderers[1])
    .setType(Renderer.MSG_SET_SURFACE)
    .send()

output log

E/EGL_emulation: eglQueryContext 32c0  EGL_BAD_ATTRIBUTE
E/EGL_emulation: tid 5679: eglQueryContext(1903): error 0x3004 (EGL_BAD_ATTRIBUTE)

I thought it was necessary to clear the surface, so I called player.clearVideoSurface, but it didn't work.
Is there any way to solve it?

[tonihei]

You probably need to set the actual surface with setPayload in the message. Not passing in the surface as payload means the video renderer detects that you want to clear the surface, that's why nothing is displayed. Please also have a look at the code of SimpleExoPlayer where this is done for the existing convenience methods:

ExoPlayer/library/core/src/main/java/com/google/android/exoplayer2/SimpleExoPlayer.java

Line 1863 in 4364b91

.setPayload(surface)

Also note that if you swap surfaces, the remove call (with null as payload) should use blockUntilDelivered on the message to ensure there is no overlap in the Surface usage. Again, please see the SimpleExoPlayer code.

[tdc-w]

Thank you for advice.
I don't understand the processing around the payload, so I will try it by checking the SimpleExoPlayer code.

[tonihei]

Closing for now as issue has been discussed and no further questions came up.