Advanced device-independent HDR tone mapping in ExoPlayer (as present e.g. in MPV)

[jan-kleks]

[REQUIRED] Use case description

Some devices (especially projectors) need tone mapping for HDR content to look good. That is why, it might be worth to explore how ExoPlayer handles HDR videos on SDR displays, and how it could be improved.

Proposed solution

There have been recently quite a few advancements in the open-source video player MPV when it comes to HDR tone mapping. It could be a good idea to look into MPV's implementation of this feature set, as such functionality would IMHO be greatly appreciated by the ExoPlayer users as well.

Apart from stock MPV, see also hdr-toys: https://github.com/natural-harmonia-gropius/hdr-toys

[dway123]

ExoPlayer doesn't currently use this, but maybe it could be helpful to use something like ExoPlayer.setVideoEffects to tone-map HDR to SDR on these devices. This can make use of OpenGL tone-mapping, which is relatively well-defined and should work at least down to API 29 (code). OpenGL tone-mapping might not have the ideal tone-mapping algorithms, but these algorithms can also be improved in the future.

[christosts]

@dway123 is tone mapping handled automatically when effects are enabled? Or does the app have to configure the effects in a specific way?

[dway123]

Oops thanks @christosts , Transformer applies tone-mapping automatically, if encoders don't support FEATURE_HdrEditing. For ExoPlayer, the output color is only automatically updated to convert HLG to PQ, but not HDR to SDR, because for ExoPlayer, HDR output is ~always supported as long as decoding is supported.

Therefore, the app does need to configure the effects in a specific way. To do this, you'll need to set up a custom VideoFrameProcessor.Factory that overrides the create method of DefaultVideoFrameProcessor.Factory's create() method, to force outputColorInfo to be ColorInfo.COLOR_TRANSFER_GAMMA_2_2. This factory would be input into a custom MediaCodecVideoRenderer (other values could be default or passthrough values), which could be output in an override of RenderersFactory#buildVideoRenderers() in a CustomRenderersFactory extends DefaultRenderersFactory. This RendererFactory could then be input into ExoPlayer via ExoPlayer.Builder.setRenderersFactory

[dway123]

This is possible, per the prior comment from Nov27.

I've created a github gist that better demonstrates how you could set up this code (based on the ExoPlayer demo), here: https://gist.github.com/dway123/934126c050482e9a2d3dcd43934b3730

[dway123]

This admittedly is a bit complicated to set up at the moment, because we didn't think this was too common of a codepath. If many folks use this, we may want to consider making a simpler codepath. One way to do this could be to implement a sort of color management in our VideoFrameProcessor pipeline (knowing input/output color combos), and allow tone-mapping to be an injectable GlEffect

If we do this, users could simply do OpenGL tonemapping by calling something like exoplayer.setVideoEffects(ImmutableList.of(new ToneMapToSdrEffect())), instead of the roughly 77 lines of mostly boilerplate code in the prior comment's github gist

[dway123]

Closing as fixed, as this is now possible, per the last two comments