Support for custom actions

[spun]

In androidx.media we could add custom actions (action + name + icon) to the PlaybackStateCompat, and the actions added were displayed in UIs like Android Auto.

I've seen references to custom SessionCommands, but PlayerWrapper.createPlaybackStateCompat doesn't use them to create the PlaybackStateCompat, so I assume they serve a different purpose.

Are custom actions supported in media3?

[marcbaechinger]

Thanks for the question!

Yes, AndroidX Media3 will support custom actions. In general, AndroidX Media3 needs to support Notifications, Auto, Automotive OS, WearOS like all the features you can do with androidx.media. We will migrate UAMP to Media3 which includes all these features but we are not there yet. I will update this issue when we have created a first version of UAMP that will probably be released in a branch of the project on GitHub first.

[marcbaechinger]

This landed with this commit in the main branch and will be part of the next release 8d03fdfe3464f073bd6b24ef3b5d2b42f6d49a39.

Supporting custom actions for Android Auto, Wear OS and the System UI media notification involves three steps:

1. Declare your custom commands in the available session commands when a controller connects:

override fun onConnect(
      session: MediaSession,
      controller: MediaSession.ControllerInfo
    ): MediaSession.ConnectionResult {
      val connectionResult = super.onConnect(session, controller)
      val availableSessionCommands = connectionResult.availableSessionCommands.buildUpon()
      customCommands.forEach { commandButton ->
        // Add custom command to available session commands.
        commandButton.sessionCommand?.let { availableSessionCommands.add(it) }
      }
      return MediaSession.ConnectionResult.accept(
        availableSessionCommands.build(),
        connectionResult.availablePlayerCommands
      )
    }

2. Send the custom layout to the controllers in onPostConnect:

override fun onPostConnect(session: MediaSession, controller: MediaSession.ControllerInfo) {
      if (!customLayout.isEmpty()) {
        // Let the controller now about the custom layout right after it connected.
        mediaLibrarySession.setCustomLayout(controller, customLayout)
      }
    }

3. Handle custom actions coming from the notification as session commands or from PendingIntents. It doesn't matter whether a command is coming from a PendingIntent of a custom notificaiton actions, a custom legacy session action or Media3 controller custom actions. All messages arrive at MediaSession.Callback.onCustomCommand(session: MediaSession, controller: MediaSession.ControllerInfo, customCommand: SessionCommand, args: Bundle):

override fun onCustomCommand(
      session: MediaSession,
      controller: MediaSession.ControllerInfo,
      customCommand: SessionCommand,
      args: Bundle
    ): ListenableFuture<SessionResult> {
      if (CUSTOM_COMMAND_TOGGLE_SHUFFLE_MODE_ON == customCommand.customAction) {
        // Enable shuffling.
        player.shuffleModeEnabled = true
        // Change the custom layout to contain the `Disable shuffling` command.
        customLayout = ImmutableList.of(customCommands[1])
        // Send the updated custom layout to controllers.
        session.setCustomLayout(customLayout)
      } else if (CUSTOM_COMMAND_TOGGLE_SHUFFLE_MODE_OFF == customCommand.customAction) {
        // Disable shuffling.
        player.shuffleModeEnabled = false
        // Change the custom layout to contain the `Enable shuffling` command.
        customLayout = ImmutableList.of(customCommands[0])
        // Send the updated custom layout to controllers.
        session.setCustomLayout(customLayout)
      }
      return Futures.immediateFuture(SessionResult(SessionResult.RESULT_SUCCESS))
    }

4. Set the custom layout right after session creation

// called from onCreate()
mediaLibrarySession =
      MediaLibrarySession.Builder(this, player, librarySessionCallback)
         [...]
        .build()
// Send custom layout to legacy session.
mediaLibrarySession.setCustomLayout(customLayout)

If handling a custom commands requires to change the actions, you need to update the custom layout and send it to the controllers with session.setCustomLayout(customLayout) to change the buttons appearing in the notification (see sample snippet above).

[spun]

Thanks for the implementation and the detailed steps!

I've created a simple custom action following the steps and have a question and some feedback.

If I control my app with the MediaControllerTester app, the list of "App-provided Custom Actions" is always empty.

I've noticed that MediaSessionImpl has two setCustomLayout, setCustomLayout(controller, customLayout) and setCustomLayout(customLayout), but only the second one calls playerWrapper.setCustomLayout. I can confirm that when I added a call to the second one, the custom action appeared in the MediaControllerTester app.

For now I'm calling session.setCustomLayout(customCommands) inside my onConnect but it's a bit awkward that we need to remember to call it at some point to maintain compatibility.

I would like to know if my call to session.setCustomLayout(customCommands) inside onConnect is a good approach or if I'm missing something to fix the PlaybackStateCompat.

---

About the implementation, I find the step of defining the sessionCommands inside onConnect to then notify the controller onPostConnect that a subsection of the available sessionCommands are actually CommandButtons seems a bit complicated.

In my opinion, the MediaSession.ConnectionResult.accept returned by onConnect could receive a list of CommandButtons. MediaLibraryService could be the one doing the transformations to sessionCommnands and managing the onPostConnect for the developer to avoid mistakes.

On the other hand, it would be helpful if we were able to access the CommandButtons directly from the controller and leave the onSetCustomLayout listener for updates to the layout/commands. Being able to see the custom action when querying the available sessionCommands but needing to add the listener to know more about the command (display name, icon, etc) is a bit more work than expected. After seeing how setCustomLayout works, I know this is way more complicated to add than it seems an probably not worth the effort, it's just something I noticed on my first aproach.

I hope that my comment didn't come across as ungrateful or whiny, I'm very very happy to have this feature, it's just that CustomActions are going to be important in Android 13 and I think some small changes could help developers, especially after seeing how media3 made everything media related intuitive for me.

Thanks again for adding the support for custom actions!

[marcbaechinger]

I hope that my comment didn't come across as ungrateful or whiny,

Totally not! Thanks for the feedback please keep it coming!

I have added the required steps to the demo app and this will be synced start of next week so you can see everything in detail (you got it right though already).

What you describe is pretty much the plan for a future version. We need to have the public API ready for Beta, so that is how we go for now. We can make things easier later by keeping the same API by sending the custom layout with the initial state as you describe.

The basic asymmetry this comes from is that Media3 controllers have a state on it's own while all legacy controller share the same state (stored in the platform session). It is possible to send different custom layouts to different Media3 controllers, while all legacy controllers share the same custom actions of the legacy session.

The current solution is sending the custom layout to Media3 controllers in a fire-and-forget way, because the custom layout is not yet part of the state of a Media3 controller. There are plans to do this in a future version. If we integrate the custom layout into the Media3 controller state, the step in onPostConnect can be made obsolete which is obviously easier for apps.

Some details:

You need to call session.setCustomLayout(customCommands) once right after you have created the session.

// called from onCreate()
mediaLibrarySession =
      MediaLibrarySession.Builder(this, player, librarySessionCallback)
         [...]
        .build()
// Send custom layout to legacy session.
mediaLibrarySession.setCustomLayout(customLayout)

This send the custom layout to all connected Media3 controllers at that moment (probably none right after session creation) and it sends it to the legacy session. Later you need to send the custom layout to all connecting Media3 controllers. The legacy controllers get their custom actions from the legacy session, so in onPostConnect you only need to send to Media3 controllers to be precise (note: The DefaultMediaNotificationProvider receives the custom layout as a Media3 controller).

I would like to know if my call to session.setCustomLayout(customCommands) inside onConnect
is a good approach or if I'm missing something to fix the PlaybackStateCompat.

Strictly speaking you should only send to a specific controller in onPostConnect and it's only required for Media3 controllers. Something along these lines:

override fun onPostConnect(session: MediaSession, controller: ControllerInfo) {
      if (!customLayout.isEmpty() && controller.controllerVersion != 0) {
        // Let Media3 controller (for instance the MediaNotificationProvider) know about the custom
        // layout right after it connected.
        ignoreFuture(mediaLibrarySession.setCustomLayout(controller, customLayout))
      }
    }