Simplify/allow selection of multiple renderers of the same type in DefaultTrackSelector.

[mrj976]

I want play multi sound in same time(simultaneously) in exoplayer but it only play mediaSource1 and mediaSource2. I want play all mediaSources at the same time(simultaneously)

My code with exoplayer is:

 Uri uri1 = Uri.parse(videopath);
 Uri uri2 = Uri.parse(audio_a.getAbsolutePath());
 Uri uri3 = Uri.parse(audio_b.getAbsolutePath());
 Uri uri4 = Uri.parse(audio_c.getAbsolutePath());

MediaSource mediaSource1 = buildMediaSource(uri1);
MediaSource mediaSource2 = buildMediaSource(uri2);
MediaSource mediaSource3 = buildMediaSource(uri3);
MediaSource mediaSource4 = buildMediaSource(uri4);

MergingMediaSource mediaSource = new MergingMediaSource(mediaSource1, mediaSource2, mediaSource3, mediaSource4);

exoPlayer.prepare(mediaSource, false, false);

please help me
thanks

[tonihei]

Multiple streams with the same type of media require multiple renderers to play them at the same time. By default, ExoPlayer's DefaultRenderersFactory provides one MediaCodecAudioRenderer and potentially some extension audio renderers. So it's actually surprising that two of the audio files play in parallel already.

You could in theory provide multiple MediaCodecAudioRenderers in a RenderersFactory you pass in to ExoPlayerFactory. However, audio renderers also function as the media clock and the player fails if multiple media clocks are provided. You can circumvent this problem by subclassing MediaCodecAudioRenderer and returning null from getMediaClock. I haven't tested this, but this would probably allow the parallel playback. Note however, that there won't be any time synchronization between the renderers and the audio files may drift apart.

[tonihei]

If your app is doing nothing else with ExoPlayer and all the files are local, you can also consider using SoundPool without any ExoPlayer integration.

[mrj976]

Multiple streams with the same type of media require multiple renderers to play them at the same time. By default, ExoPlayer's DefaultRenderersFactory provides one MediaCodecAudioRenderer and potentially some extension audio renderers. So it's actually surprising that two of the audio files play in parallel already.

You could in theory provide multiple MediaCodecAudioRenderers in a RenderersFactory you pass in to ExoPlayerFactory. However, audio renderers also function as the media clock and the player fails if multiple media clocks are provided. You can circumvent this problem by subclassing MediaCodecAudioRenderer and returning null from getMediaClock. I haven't tested this, but this would probably allow the parallel playback. Note however, that there won't be any time synchronization between the renderers and the audio files may drift apart.

Thanks
I have 6 audio files
Can you send me an example?

[mrj976]

If your app is doing nothing else with ExoPlayer and all the files are local, you can also consider using SoundPool without any ExoPlayer integration.

I want play a video with 6 audio files
So I used ExoPlayer
Is there any other way to solve this problem?
Use another library

[tonihei]

The way I described above seems to work with the added complication of needing a custom TrackSelector implementation that links tracks to audio renderers.

Please note that the renderers are not actively synchronized to each other, so depending on your alignment requirements between these streams, this may be a problem. You will also need to select the video track in the track selector if you want to enable it.

Code I used for testing:

-----MediaSource-----
mediaSource = new MergingMediaSource(source1, ..., source4);

----RendersFactory-----
RenderersFactory renderersFactory = new DefaultRenderersFactory(this) {
  @Override
  protected void buildAudioRenderers(Context context, int extensionRendererMode,
      MediaCodecSelector mediaCodecSelector,
      @Nullable DrmSessionManager<FrameworkMediaCrypto> drmSessionManager,
      boolean playClearSamplesWithoutKeys, boolean enableDecoderFallback,
      AudioProcessor[] audioProcessors, Handler eventHandler,
      AudioRendererEventListener eventListener, ArrayList<Renderer> out) {
        super.buildAudioRenderers(context, extensionRendererMode, mediaCodecSelector,
          drmSessionManager, playClearSamplesWithoutKeys, enableDecoderFallback,
          audioProcessors, eventHandler, eventListener, out);
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
    }
 };

with the following class:

private static final class AudioRendererWithoutClock extends MediaCodecAudioRenderer {
    public AudioRendererWithoutClock(Context context,
        MediaCodecSelector mediaCodecSelector) {
      super(context, mediaCodecSelector);
    }
    @Override
    public MediaClock getMediaClock() {
      return null;
    }
}

---- TrackSelector ----
TrackSelector trackSelector = new TrackSelector() {
    @Override
    public TrackSelectorResult selectTracks(RendererCapabilities[] rendererCapabilities,
        TrackGroupArray trackGroups, MediaPeriodId periodId, Timeline timeline)
        throws ExoPlaybackException {
      Queue<Integer> audioRenderers = new ArrayDeque<>();
      RendererConfiguration[] configs = new RendererConfiguration[rendererCapabilities.length];
      TrackSelection[] selections = new TrackSelection[rendererCapabilities.length];
      for (int i = 0; i < rendererCapabilities.length; i++) {
        if(rendererCapabilities[i].getTrackType() == C.TRACK_TYPE_AUDIO) {
          audioRenderers.add(i);
        }
      }
      for (int i = 0; i < trackGroups.length; i++) {
        if (MimeTypes.isAudio(trackGroups.get(i).getFormat(0).sampleMimeType)) {
          Integer index = audioRenderers.poll();
          if (index != null) {
            selections[index] = new FixedTrackSelection(trackGroups.get(i), 0);
            configs[index] = RendererConfiguration.DEFAULT;
          }
        }
      }
      return new TrackSelectorResult(configs, selections, new Object());
    }

    @Override
    public void onSelectionActivated(Object info) {
     }
 };

---- Player ----
player =
      new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
         .setTrackSelector(trackSelector)
         .build();
player.prepare(mediaSource);

[mrj976]

The way I described above seems to work with the added complication of needing a custom TrackSelector implementation that links tracks to audio renderers.

Please note that the renderers are not actively synchronized to each other, so depending on your alignment requirements between these streams, this may be a problem. You will also need to select the video track in the track selector if you want to enable it.

Code I used for testing:

-----MediaSource-----
mediaSource = new MergingMediaSource(source1, ..., source4);

----RendersFactory-----
RenderersFactory renderersFactory = new DefaultRenderersFactory(this) {
  @Override
  protected void buildAudioRenderers(Context context, int extensionRendererMode,
      MediaCodecSelector mediaCodecSelector,
      @Nullable DrmSessionManager<FrameworkMediaCrypto> drmSessionManager,
      boolean playClearSamplesWithoutKeys, boolean enableDecoderFallback,
      AudioProcessor[] audioProcessors, Handler eventHandler,
      AudioRendererEventListener eventListener, ArrayList<Renderer> out) {
        super.buildAudioRenderers(context, extensionRendererMode, mediaCodecSelector,
          drmSessionManager, playClearSamplesWithoutKeys, enableDecoderFallback,
          audioProcessors, eventHandler, eventListener, out);
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
    }
 };

with the following class:

private static final class AudioRendererWithoutClock extends MediaCodecAudioRenderer {
    public AudioRendererWithoutClock(Context context,
        MediaCodecSelector mediaCodecSelector) {
      super(context, mediaCodecSelector);
    }
    @Override
    public MediaClock getMediaClock() {
      return null;
    }
}

---- TrackSelector ----
TrackSelector trackSelector = new TrackSelector() {
    @Override
    public TrackSelectorResult selectTracks(RendererCapabilities[] rendererCapabilities,
        TrackGroupArray trackGroups, MediaPeriodId periodId, Timeline timeline)
        throws ExoPlaybackException {
      Queue<Integer> audioRenderers = new ArrayDeque<>();
      RendererConfiguration[] configs = new RendererConfiguration[rendererCapabilities.length];
      TrackSelection[] selections = new TrackSelection[rendererCapabilities.length];
      for (int i = 0; i < rendererCapabilities.length; i++) {
        if(rendererCapabilities[i].getTrackType() == C.TRACK_TYPE_AUDIO) {
          audioRenderers.add(i);
          configs[i] = RendererConfiguration.DEFAULT;
        }
      }
      for (int i = 0; i < trackGroups.length; i++) {
        if (MimeTypes.isAudio(trackGroups.get(i).getFormat(0).sampleMimeType)) {
          Integer index = audioRenderers.poll();
          if (index != null) {
            selections[index] = new FixedTrackSelection(trackGroups.get(i), 0);
          }
        }
      }
      return new TrackSelectorResult(configs, selections, new Object());
    }

    @Override
    public void onSelectionActivated(Object info) {
     }
 };

---- Player ----
player =
      new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
         .setTrackSelector(trackSelector)
         .build();
player.prepare(mediaSource);

Thank you
I used your code but there is error in this section:

player =
      new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
         .setTrackSelector(trackSelector)
         .build();

In above code ".Builder" is red and show this message: " Cannot resolve symbol 'Builder' "

In my MediaSources:
mediasource1 is video file
mediasource2 , mediasource3 , mediasource4, mediasource5, mediasource6, mediasource7 are audio files

what should I do?

[tonihei]

The builder is only available in the latest ExoPlayer version. You can use ExoPlayerFactory instead where you can set your own track selector and renderers factory.

[mrj976]

The builder is only available in the latest ExoPlayer version. You can use ExoPlayerFactory instead where you can set your own track selector and renderers factory.

Thank you for your sample code and your help
I used exoplayer with 2.10.7 version and its latest version of exoplayer. but show this message: " Cannot resolve symbol 'Builder' " again.
implementation 'com.google.android.exoplayer:exoplayer:2.10.7'
I solved this error with this code:
Player = ExoPlayerFactory.newSimpleInstance(this, renderersFactory, trackSelector);

with your code i can play multiple audio files but dont show video.

My code in media source dection:

Uri uri1 = Uri.parse(video_path);
        Uri uri2 = Uri.parse(audio_path1);
        Uri uri3 = Uri.parse(audio_path2);
        Uri uri4 = Uri.parse(audio_path3);

        MediaSource mediaSource1 = buildMediaSource(uri1);
        MediaSource mediaSource2 = buildMediaSource(uri2);
        MediaSource mediaSource3 = buildMediaSource(uri3);
        MediaSource mediaSource4 = buildMediaSource(uri4);

        MergingMediaSource mediaSource = new MergingMediaSource(mediaSource1, mediaSource2, mediaSource3, mediaSource4);

what should I do for play video with this audio files in same time?

[tonihei]

I used exoplayer with 2.10.7 version and its latest version of exoplayer. but show this message: " Cannot resolve symbol 'Builder' " again.

Yes, sorry, this will only be part of 2.11.0 (once released). Your alternative is correct.

i can play multiple audio files but dont show video

As pointed out above, you also need to select the video track in the TrackSelector implementation. Search for a renderer index that supports C.TRACK_TYPE_VIDEO and find a track where MimeTypes.isVideo returns true, and then create a new FixedTrackSelection similar to what I've done with the audio tracks in my example.

Note that writing you own TrackSelector is quite an advanced customization, so I won't dive any deeper with code examples.

complication of needing a custom TrackSelector

This turned out to be quite complicated and I'll use this issue to track an enhancement for DefaultTrackSelector that allows selecting multiple renderers of the same type (e.g. multiple video outputs, multiple audio renderers, etc.). But it's low-priority for now.

[wangxuan217]

The way I described above seems to work with the added complication of needing a custom TrackSelector implementation that links tracks to audio renderers.
Please note that the renderers are not actively synchronized to each other, so depending on your alignment requirements between these streams, this may be a problem. You will also need to select the video track in the track selector if you want to enable it.
Code I used for testing:

-----MediaSource-----
mediaSource = new MergingMediaSource(source1, ..., source4);

----RendersFactory-----
RenderersFactory renderersFactory = new DefaultRenderersFactory(this) {
  @Override
  protected void buildAudioRenderers(Context context, int extensionRendererMode,
      MediaCodecSelector mediaCodecSelector,
      @Nullable DrmSessionManager<FrameworkMediaCrypto> drmSessionManager,
      boolean playClearSamplesWithoutKeys, boolean enableDecoderFallback,
      AudioProcessor[] audioProcessors, Handler eventHandler,
      AudioRendererEventListener eventListener, ArrayList<Renderer> out) {
        super.buildAudioRenderers(context, extensionRendererMode, mediaCodecSelector,
          drmSessionManager, playClearSamplesWithoutKeys, enableDecoderFallback,
          audioProcessors, eventHandler, eventListener, out);
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
        out.add(new AudioRendererWithoutClock(context, mediaCodecSelector));
    }
 };

with the following class:

private static final class AudioRendererWithoutClock extends MediaCodecAudioRenderer {
    public AudioRendererWithoutClock(Context context,
        MediaCodecSelector mediaCodecSelector) {
      super(context, mediaCodecSelector);
    }
    @Override
    public MediaClock getMediaClock() {
      return null;
    }
}

---- TrackSelector ----
TrackSelector trackSelector = new TrackSelector() {
    @Override
    public TrackSelectorResult selectTracks(RendererCapabilities[] rendererCapabilities,
        TrackGroupArray trackGroups, MediaPeriodId periodId, Timeline timeline)
        throws ExoPlaybackException {
      Queue<Integer> audioRenderers = new ArrayDeque<>();
      RendererConfiguration[] configs = new RendererConfiguration[rendererCapabilities.length];
      TrackSelection[] selections = new TrackSelection[rendererCapabilities.length];
      for (int i = 0; i < rendererCapabilities.length; i++) {
        if(rendererCapabilities[i].getTrackType() == C.TRACK_TYPE_AUDIO) {
          audioRenderers.add(i);
          configs[i] = RendererConfiguration.DEFAULT;
        }
      }
      for (int i = 0; i < trackGroups.length; i++) {
        if (MimeTypes.isAudio(trackGroups.get(i).getFormat(0).sampleMimeType)) {
          Integer index = audioRenderers.poll();
          if (index != null) {
            selections[index] = new FixedTrackSelection(trackGroups.get(i), 0);
          }
        }
      }
      return new TrackSelectorResult(configs, selections, new Object());
    }

    @Override
    public void onSelectionActivated(Object info) {
     }
 };

---- Player ----
player =
      new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
         .setTrackSelector(trackSelector)
         .build();
player.prepare(mediaSource);

Thank you
I used your code but there is error in this section:

player =
      new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
         .setTrackSelector(trackSelector)
         .build();

In above code ".Builder" is red and show this message: " Cannot resolve symbol 'Builder' "

In my MediaSources:
mediasource1 is video file
mediasource2 , mediasource3 , mediasource4, mediasource5, mediasource6, mediasource7 are audio files

what should I do?

Can I switch between the video track and the audio track?

[ziad-halabi9]

I had the same code suggested here in order to play multiple tracks at the same time. But now after updating to ExoPlayer 2.17, it stopped working. The following is the error I can see:

E/ExoPlayerImplInternal: Playback error
      com.google.android.exoplayer2.ExoPlaybackException: Unexpected runtime error
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:624)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:237)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: java.lang.IllegalStateException
        at com.google.android.exoplayer2.util.Assertions.checkState(Assertions.java:84)
        at com.google.android.exoplayer2.source.ProgressiveMediaPeriod.selectTracks(ProgressiveMediaPeriod.java:283)
        at com.google.android.exoplayer2.source.MergingMediaPeriod.selectTracks(MergingMediaPeriod.java:153)
        at com.google.android.exoplayer2.source.MaskingMediaPeriod.selectTracks(MaskingMediaPeriod.java:186)
        at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:296)
        at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:259)
        at com.google.android.exoplayer2.MediaPeriodHolder.handlePrepared(MediaPeriodHolder.java:193)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handlePeriodPrepared(ExoPlayerImplInternal.java:2247)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:517)
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:237) 
        at android.os.HandlerThread.run(HandlerThread.java:67) 

Any suggestions on what can be causing this and if there's a quick fix?

[Brutalnung]

Thanks ในวันที่ พฤ. 10 มี.ค. 2022 10:54 น. ziad-halabi9 ***@***.***> เขียนว่า:

…

I had the same code suggested here in order to play multiple tracks at the same time. But now after updating to ExoPlayer 2.17, it stopped working. The following is the error I can see: E/ExoPlayerImplInternal: Playback error com.google.android.exoplayer2.ExoPlaybackException: Unexpected runtime error at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:624) at android.os.Handler.dispatchMessage(Handler.java:103) at android.os.Looper.loop(Looper.java:237) at android.os.HandlerThread.run(HandlerThread.java:67) Caused by: java.lang.IllegalStateException at com.google.android.exoplayer2.util.Assertions.checkState(Assertions.java:84) at com.google.android.exoplayer2.source.ProgressiveMediaPeriod.selectTracks(ProgressiveMediaPeriod.java:283) at com.google.android.exoplayer2.source.MergingMediaPeriod.selectTracks(MergingMediaPeriod.java:153) at com.google.android.exoplayer2.source.MaskingMediaPeriod.selectTracks(MaskingMediaPeriod.java:186) at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:296) at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:259) at com.google.android.exoplayer2.MediaPeriodHolder.handlePrepared(MediaPeriodHolder.java:193) at com.google.android.exoplayer2.ExoPlayerImplInternal.handlePeriodPrepared(ExoPlayerImplInternal.java:2247) at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:517) at android.os.Handler.dispatchMessage(Handler.java:103) at android.os.Looper.loop(Looper.java:237) at android.os.HandlerThread.run(HandlerThread.java:67) Any suggestions on what can be causing this and if there's a quick fix? — Reply to this email directly, view it on GitHub <#6589 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWY6HDK3XGTR67ZSANJGR33U7ILMNANCNFSM4JFSEOEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Brutalnung]

E/ExoPlayerImplInternal: Playback error
com.google.android.exoplayer2.ExoPlaybackException: Unexpected runtime error
at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:624)
at android.os.Handler.dispatchMessage(Handler.java:103)
at android.os.Looper.loop(Looper.java:237)
at android.os.HandlerThread.run(HandlerThread.java:67)
Caused by: java.lang.IllegalStateException
at com.google.android.exoplayer2.util.Assertions.checkState(Assertions.java:84)
at com.google.android.exoplayer2.source.ProgressiveMediaPeriod.selectTracks(ProgressiveMediaPeriod.java:283)
at com.google.android.exoplayer2.source.MergingMediaPeriod.selectTracks(MergingMediaPeriod.java:153)
at com.google.android.exoplayer2.source.MaskingMediaPeriod.selectTracks(MaskingMediaPeriod.java:186)
at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:296)
at com.google.android.exoplayer2.MediaPeriodHolder.applyTrackSelection(MediaPeriodHolder.java:259)
at com.google.android.exoplayer2.MediaPeriodHolder.handlePrepared(MediaPeriodHolder.java:193)
at com.google.android.exoplayer2.ExoPlayerImplInternal.handlePeriodPrepared(ExoPlayerImplInternal.java:2247)
at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:517)
at android.os.Handler.dispatchMessage(Handler.java:103) 
at android.os.Looper.loop(Looper.java:237) 
at android.os.HandlerThread.run(HandlerThread.java:67) 

[Brutalnung]

ขอคำชี้แจง