Compatibility with WebTransport

[wilaw]

WebTransport is a framework which enables data transfer between browsers and servers with support for multiple data flows, unidirectional data flows, out-of-order delivery, variable reliability and pluggable protocols. The WebTransport spec is being developed at the IETF and in parallel the W3C just chartered a new work group to address a browser API. A candidate API has been incubated by the WICG.

The request is for MSE v2 to smoothly integrate with WebTransport API, specifically with support for binding the output of a ReadableStream to the input of a source buffer.

[chrisn]

See related issue: #14

[wolenetz]

Beyond an API like appendStream (#14), is there any requirement for additional signalling capabilities needed in hypothetical extension of MSE to support WebTransport?

[aboba]

One of the scenarios for WebTransport is "low latency streaming". To support this scenario in an interoperable way across browsers, MSE v2 would need to standardize "Low Latency MSE", which is poorly documented today.

Firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1520894

Peter Thatcher covered some of the issues in these presentations:
https://docs.google.com/presentation/d/1_xQSoIdN-srjBc-GE_vuQMkxkaer2G-mZBchLPTyY20/
https://www.youtube.com/watch?reload=9&v=VD5GBLBiSxo

[wolenetz]

@#256 (comment): Thank you for providing those links.
There are several related efforts either in MSEv2 consideration or in other concurrent spec incubations to better support low-latency media playback scenarios:

• HTMLMediaElement.latencyHint: Add hint attribute to HTMLMediaElement to configure rendering latency whatwg/html#4638 is the current effort being incubated. It is not MSE-specific. Right now, the HTMLMediaElement.latencyHint attribute is in test in Chromium as incubation proceeds. The idea is that it would override any result of the implementation's "live/low-delay" detection heuristic, giving the application more control.

• Eviction policies (such as allowing an MSE mode whereby an "infinite length" GOP could be buffered, with caveat that seeking would be disallowed in the extended HTMLMediaElement when using that particular eviction policy). Define a preemptive eviction MSE API for low-memory platforms #232 has more information.

• WebCodecs: https://github.com/WICG/web-codecs -- lets the app use the platform encoders and decoders for audio/video. Caveat is the app must also handle higher level functions (such as what MSE provides at a middle layer currently) to support canonical media buffering and playback. However, there are strong use case overlaps between MSE and WebCodecs in the live, low-latency space.

• MSE-in-Workers: (MediaSource in Worker #175): This could help reduce buffering-starvation-induced playback "jank" when the main thread is too busy to feed the MSE API with media to buffer. While not solely for live/low-latency, MediaSource in Worker #175 could contribute to reducing the buffering stalls which could otherwise prevent the playback from maintaining low-latency.

Note that I do not construe any of these efforts currently as invalidating MSE feature requests for appendStream (#14) nor this issue (#256).