Media Source Extensions for WebCodecs

[wolenetz]

HIQaH! QaH! TAG!

I'm requesting a TAG review of Media Source Extensions for WebCodecs.

The Media Source Extensions API (MSE) requires applications to provide fragments of containerized media (such as fragmented MP4, WebM, or MP3) to be able to buffer and play that media. If the application already has the media in a parsed and structured form, it can cost extra latency and code complexity to package that media into a container and feed it to MSE. As the web platform is evolving to give applications lower-level abstractions for efficiently encoding and decoding media via the WebCodecs API, MSE can use WebCodec's media structures to let applications more efficiently feed their media to MSE. We propose additional MSE methods to provide web authors easy ability to buffer "containerless" media as an alternative to existing MSE methods that required containerized, "muxed" media.

• Explainer (minimally containing user needs and example code): https://github.com/wolenetz/mse-for-webcodecs/blob/main/explainer.md
• Security and Privacy self-review: https://github.com/wolenetz/mse-for-webcodecs/blob/main/security-privacy-questionnaire.md
• Primary contacts (and their relationship to the specification):
  • Matt Wolenetz (@wolenetz, Google) - MSE spec co-editor and editor of this new feature.
• Organization/project driving the design: Google Chrome
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status):
  • Chrome Status: https://www.chromestatus.com/features/5649291471224832
  • MSE specification issue: Enable buffering of WebCodecs Encoded Chunks for playback with MSE - aka "MSE for WebCodecs" or "MSE4WC" w3c/media-source#184 (comment)
  • Chromium feature bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1144908

Further details:

• I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): Media WG
• The group where standardization of this work is intended to be done ("unknown" if not known): Media WG
• Major unresolved issues with or opposition to this design: None. Minor open questions are listed in the Explainer's Open Questions section

You should also know that...

• The review is early in the prototyping and design stage because we expect to learn from early implementation feedback more precise answers to the Explainer's Open Questions section, so we'd like to get a rough design reviewed to help refine eventual implementation and specification for this feature.

We'd prefer the TAG provide feedback as:

💬 leave review feedback as a comment in this issue and @-notify @wolenetz

[cynthia]

As it looks like WebCodecs is going through some design changes, we'd like to hold off on providing feedback until that part has been ossified.

[dalecurtis]

The EncodedAudioChunk/EncodedVideoChunk parts of the WebCodecs spec aren't in flux -- which are the only parts this depends on -- so this should be safe to look at.

[cynthia]

Good to know! Thanks. Will take a look.

[cynthia]

@ylafon and I discussed this today.

We think the use-cases and the proposed API design is fine. There seems to be a bit of unanswered questions in terms of implementation details (e.g. MSE is hardware decoded, WebCodecs should also be hardware decoded, what if you don't have enough decoder capacity?) and whether or not this is a proper solution to the use-case at hand. However, we'll leave that discussion up to the group.

For now, we're happy to see this move forward. Thanks for bringing this to our attention!

[dalecurtis]

@cynthia Glad to hear! Can you elaborate on your questions though? I don't think those questions are relevant to MSE. MSE makes no guarantees about whether media is hardware or software decoded or even anything about how the underlying decoder configurations are managed (intentionally so to allow user agent management of such resources).

The proposed mechanism is only about allowing authors to append already demuxed data into MSE, thereby skipping the demuxing which normally happens implicitly through the byte based append methods. So it's a strict cost reduction mechanism with otherwise no impacts on how MSE functions.

Did y'all have any feedback on whether these new append methods should be added as promise based? Should the interface use the non-promise style (updating, updateend events) for consistency until byte-based promise methods are available? Should we only add the promise based method for chunks once we have a promise based method for bytes?

[wolenetz]

Raising these questions again here, as we haven't yet received a response about them:

Did y'all have any feedback on whether these new append methods should be added as promise based? Should the interface use the non-promise style (updating, updateend events) for consistency until byte-based promise methods are available? Should we only add the promise based method for chunks once we have a promise based method for bytes?

[cynthia]

Apologies for the delay.

Did y'all have any feedback on whether these new append methods should be added as promise based? Should the interface use the non-promise style (updating, updateend events) for consistency until byte-based promise methods are available? Should we only add the promise based method for chunks once we have a promise based method for bytes?

@plinss, @atanassov and I discussed during our VF2F today. And one of the concerns we had were if promises would provide the latency guarantees that the downstream application needs (given that current MSE is event driven, it should be okay?), and if a domain specific application of promises would make it painful to upgrade when byte-based promises are available. We don't have a strong preference for either, but we'd like to hear your thoughts on this.

Does the current design propose a generic path forward for a byte-based promise API? If so, we think promises are probably the right answer.

[wolenetz]

Did y'all have any feedback on whether these new append methods should be added as promise based? Should the interface use the non-promise style (updating, updateend events) for consistency until byte-based promise methods are available? Should we only add the promise based method for chunks once we have a promise based method for bytes?

@plinss, @atanassov and I discussed during our VF2F today. And one of the concerns we had were if promises would provide the latency guarantees that the downstream application needs (given that current MSE is event driven, it should be okay?),

Since the current MSE is event driven, I don't expect worsened latency due to apps instead awaiting Promise resolution/rejection via this new promise-based method for appending chunks versus what that latency would have been had it been event driven chunk-appending instead. In passing, I've heard that at least some MSE API users have already wrapped the existing event-driven byte-based append operations inside promises in their applications.
Note also that the ergonomics of a promise-based API are simpler, too: the extra enqueueing of 'updatestart', and 'update' events are not done (nor 'abort' or 'error' in those rejection paths). It is unclear if existing API users depend on those events, or if their absence is negligible.

and if a domain specific application of promises would make it painful to upgrade when byte-based promises are available. We don't have a strong preference for either, but we'd like to hear your thoughts on this.

I'm not sure I understand this portion of your response. If and when Promisified versions of MSE appendBuffer() is available, it's unlikely that the event-driven versions would be deprecated. Hence, upgrading from event-driven to promisified byte-based append would not necessarily require urgent prioritization by the MSE API user, though (apart from the lack of probably superfluous extra events) there would not be a big change of actual functionality made available by the Promisified versions of those methods. I feel like I'm missing your point though.

Does the current design propose a generic path forward for a byte-based promise API? If so, we think promises are probably the right answer.

The path taken in MSE-for-WebCodecs' appendEncodedChunks() lays out a pattern that indeed could be followed by a Promisified version of the existing event-driven byte-based appendBuffer(): the draft specification PR (w3c/media-source#302) for the Promise portion of the new appendEncodedChunks() reuses much of the existing state machine structure: at most one asynchronous operation can be in-flight at a time for a particular SourceBuffer, with well-defined start/complete/error/abort paths for the existing event-driven async operation (and now also for the Promisified appendEncodedChunks() async operation.

[cynthia]

Thank you for the response. To be upfront, from a review point-of-view this is complete - and we are happy to see this proposal move forward. Thank you for bringing this to us and apologies for the process delay.

Hence, upgrading from event-driven to promisified byte-based append would not necessarily require urgent prioritization by the MSE API user, though (apart from the lack of probably superfluous extra events) there would not be a big change of actual functionality made available by the Promisified versions of those methods. I feel like I'm missing your point though.

The question/concern here was whether or not the interoperability with generic byte-base promises (when it happens) can be a potential future pain point if this is too domain-specific, and the question was about whether or not there ideas have been exchanged around that. I think the response addresses some of the questions we had, although I'll have to check with the other reviewers to be sure.