Specify mixed encrypted/unencrypted content

[baconz]

There should be a way to (1) check if the browser supports and (2) specify that a stream contains a mixture of encrypted and unencrypted content. This is particularly important in the case of dynamic ad insertion, where a manifest/playlist may point to content from an out-of-band ad server that knows nothing of the encryption scheme used for the base content.

It may also be useful to specify that a stream will contain mixed codecs, but I'm not sure if that belongs in the EME.

See: https://bugs.chromium.org/p/chromium/issues/detail?id=597443

[mwatson2]

As the specification currently stands, attaching a MediaKeys object after playback has started is not guaranteed to be supported (i.e. is not supported in general):

Support for clearing or replacing the associated MediaKeys object during playback is a quality of implementation issue. In many cases it will result in a bad user experience or rejected promise.

It would be my assumption that unencrypted media can always be played - for example including after MediaKeys has been attached - but this does not appear to be required by the specification at present.

Whether a switch between two supported formats that differ in some significant way (container, codec, codec parameters, encryption etc.) is rendered seamlessly is a quality of implementation issue. Discovery of device behavior here would be in scope of MSE, but I don't believe we have that at present.

[mwatson2]

Also, see #197.

[paulbrucecotton]

It may also be useful to specify that a stream will contain mixed codecs, but I'm not sure if that belongs in the EME.

This request has been discussed previously. See:
https://lists.w3.org/Archives/Public/public-html-media/2015Oct/0083.html

"https://www.w3.org/Bugs/Public/show_bug.cgi?id=22135 and I believe https://www.w3.org/Bugs/Public/show_bug.cgi?id=22137 were RESOLVED as WONTFIX in MSE since the underlying request is an HTML5 feature request.

I believe https://www.w3.org/Bugs/Public/show_bug.cgi?id=21333 was lumped in the above two other bugs.

The HTML5 bug https://www.w3.org/Bugs/Public/show_bug.cgi?id=22785 has NOT yet been processed or resolved. I recommend that proponents of the requirements here approach the new Web Platform WG to escalate the feature request in 22785."

[joeyparrish]

@mwatson2, we are talking about a scenario in which MediaKeys has been attached specifically before any content is appended and before playback has started.

This scenario involves appending an init segment for encrypted content followed some time later by an init segment for unencrypted content.

It makes perfect sense for a DASH manifest to contain an encrypted Period with main content, followed by an unencrypted Period containing an ad, followed by another encrypted Period with main content.

This is admittedly an intersection of MSE and EME, as the MSE implementation must allow this appendBuffer() call with clear content after some amount of encrypted content.

If this is a quality of implementation issue, it is one that prevents a primary use case for anyone who wants to serve ads without encrypting them first.

[mwatson2]

There are several cases where the capabilities of the media element may change as a result of attaching a MediaKeys object or playing content:

• you may play clear content on a new media element and then attempt to attach a MediaKeys. This may fail. This is explicitly described in the specification.
• you may attach a MediaKeys object and then attempt to play content encoded with a codec not in the capabilities of the MediaKeySystemAccess object. This may fail. This is also explicitly described in the specification.

However, you are concerned about the case where you attach MediaKeys and then attempt to play clear content that is encoded with one of the codecs in the capabilities of the MediaKeySystemAccess object. There is nothing to say this may fail. I have some sympathy with the idea that otherwise present functionality should not just disappear in unexplained and unspecified ways. There is no procedure for "unencrypted block encountered" which might say that if unencrypted data is not supported at this time you could fail in some way.

I would be surprised if this was not supported in existing implementations. We could define a test for this. If we find diversity in what is supported, that would motivate fixing #197 soon so you can find out.

[joeyparrish]

Unencrypted content is already supported in general, since it is common to have a "clear lead" to give the client time to acquire licenses. In those cases, the unencrypted blocks are part of the same stream as later encrypted blocks, so only one initialization segment is given and it indicates encrypted content. I have seen no interop issues with that.

In the OP's scenario, though, there is a second stream which is unencrypted, and therefore a second initialization segment which indicates unencrypted content. We have seen Chrome reject this second init segment after the first one has already been appended. It would appear that the MSE implementation is the one rejecting the init segment.

[xhwang-chromium]

FWIW, the proposed spec text is as follows (04eeb40):

If MediaKeys is set before playback starts, the implementation MUST support switching between clear and encrypted streams.

NOTE: Whether the clear and encrypted streams - or even multiple streams of either type - may use different contentTypes is left to other specifications related to media sources and capability detection.

NOTE: Applications are advised that at the time of this writing, at least some major browser do not support this.

Support of switching when MediaKeys is set after the playback starts, or when the clear stream uses a different contentType, is a quality of implementation issue.

NOTE: Applications are advised that at the time of this writing, at least some major browser do not support this. Applications that need this functionality SHOULD set MediaKeys before loading the media resource.

Let's resume the discussion and reach a conclusion on this matter.

[joeyparrish]

Moved out of v2 milestone. This isn't a high enough priority to get finished in time for the next working draft.

[chrisn]

Discussed in Media WG meeting 16 May 2023. General agreement in support of the change proposed in #251 (comment), and will help with interoperability.

[xhwang-chromium]

@joeyparrish @gregwfreedman @jernoble Could you please take a look at the suggestion above and share your thoughts? Thanks!

[calvaris]

I don't think allowing using different content types for clear and encrypted content is a good idea, specially if you don't support changeType. I think changeType should be used if you want to switch content types.

[xhwang-chromium]

I don't think allowing using different content types for clear and encrypted content is a good idea, specially if you don't support changeType. I think changeType should be used if you want to switch content types.

changeType should be used when changing content types. This issue is about mixing clear and encrypted contents, regardless they have the same content type or not, so I feel it's orthogonal to the changeType API.

[calvaris]

I don't think allowing using different content types for clear and encrypted content is a good idea, specially if you don't support changeType. I think changeType should be used if you want to switch content types.

changeType should be used when changing content types. This issue is about mixing clear and encrypted contents, regardless they have the same content type or not, so I feel it's orthogonal to the changeType API.

No, it is not orthogonal. One thing is mixing encrypted and clear content that are using the same content type and another thing is mixing clear and encrypted content that use different content types. In the former I agree with you, in the latter I definitely thing we should use changeType as if we were talking about just clear (or just encrypted) content.

[xhwang-chromium]

another thing is mixing clear and encrypted content that use different content types

That's not what I meant. What I said was "this issue is about mixing clear and encrypted contents, regardless they have the same content type or not".

What you are suggesting here exactly? I feel a bit lost.

[calvaris]

I am referring to this part:

NOTE: Whether the clear and encrypted streams - or even multiple streams of either type - may use different contentTypes is left to other specifications related to media sources and capability detection.

I think we should not "leave to other specifications" if we can or can't use different content types. IMHO, the content type should be the same when switching back and forth from encrypted to unencrypted but if we decide to change the content type, a changeType must be issued (if supported).

[xhwang-chromium]

We discussed this issue and the proposal in #251 (comment) at TPAC 2024. There's no objection from the group and both EME editors are in favor of the change.

Meeting notes are available at: https://www.w3.org/2024/09/26-mediawg-minutes.html#nt02

Next we need to figure out how to merge this into the EME spec.

[chrisn]

A few comments, having reviewed the proposed text more closely.

1. Regarding where to change the EME spec, would this be a change to setMediaKeys?
2. For "If MediaKeys is set before playback starts" should we reference the HTML spec to precisely identify what state the media element should be in, e.g., is this dependent on its readyState?
3. What should happen if an implementation doesn't support switching between clear and encrypted streams, should setMediaKeys reject?
4. I think it may be helpful to say something more about switching content types, e.g., to point out that changeType() is needed.
5. Specs typically don't include notes about implementation status, so I suggest we don't here. The hope is that this will be an interoperable feature and will be implemented more widely eventually.