Refine srcObject's MediaSourceHandle behavior

[wolenetz]

Current description of this PR:
Overall intent for a MediaSourceHandle is that it should be
[Transferable], and be restricted to "at most one" successful attachment
to the underlying MediaSource by a handle. Further, if a Window
MediaSource was previously attached using a legacy MSE object URL,
subsequent attempts to use that MediaSource instance's
MediaSourceHandle for attachment must also fail.

• Changes MediaSource.getHandle() to be a [SameObject] readonly attribute named handle.

• Adds HTMLMediaElement.srcObject extension subsection.

• Updates MediaSourceHandle transfer and related attachment text to more
  clearly indicate that transferring a handle prevents re-transfer of
  that instance due to the [[Detached]] internal slot's logic, and that
  transfer is also synchronously prevented if the handle instance's
  [[has ever been assigned as srcobject]] internal slot is true. Also
  adds multiple notes to describe intent and to clarify.

• Updates the "attaching to a media element" text to indicate that a
  MediaSourceHandle can be attached successfully to at most one media
  element ever.
  Note that legacy MSE object URL attachments similarly already achieve
  the same intent in existing implementations. This change includes a
  clarification for MSE object URL attachments, since it was never the
  intent even for main-thread MediaSource attachments for them to be
  successfully attached to more than one HTMLMediaElement at a time.
  The switch to more clear srcObject usage for worker MSE attachment
  affords the spec an opportunity to be more clear about this intent.

This refinement originates from discussion on the previous PR with
@karlt (#305) and from lengthy
discussion on this change itself at
#306.

This work is associated with the MSE-in-Worker spec issue:
#175

Original description of this PR (updated more recently per #306 (comment)):

Adds text intending to help prevent using one or more
MediaSourceHandle for the same underlying MediaSource for
simultaneous, concurrent attachments to media elements:

• Updates MediaSource.getHandle() to succeed at most once per
  MediaSource instance.
• Updates MediaSourceHandle transfer section to more clearly indicate
  that transferring a handle out of a context makes that handle
  unavailable for transferring again out of the same context (unless it
  is first transferred back to that context).
• Updates the "attaching to a media element" text to indicate that a
  MediaSourceHandle can be attached successfully to at most one media
  element at a time, and with the previous 2 clarifications, ensures
  that the underlying worker context MediaSource can only be attached
  successfully to at most one media element at a time.

Note that legacy MSE object URL attachments similarly already achieve
the same intent in existing implementations. This change includes a
clarification for those similar to the third bullet, but for a
MediaSource in the "attaching to a media element" text, since it was
never the intent even for main-thread MediaSource attachments for them
to be successfully attached to more than one HTMLMediaElement at a
time. The switch to more clear srcObject usage for worker MSE
attachment affords the spec an opportunity to be more clear about this
intent.

This refinement originates from discussion on the previous PR with
@karlt: #305

This work is associated with the MSE-in-Worker spec issue:
#175

---

Preview | Diff

[wolenetz]

@mwatson2, please review. Also, @karlt, please take a look to ensure I've addressed the issues we discussed in #305.

Thank you!

[karlt]

I'm not clear on the best way to deal with a transfer of the MediaSourceHandle while it is set on a srcObject attribute. Alternatives to consider:

1. Allow the transfer and error on the media element.
2. Block transfer while on the srcObject attribute.
   The "transfer steps" caller is expecting a possible exception from DetachArrayBuffer(transferable) in the [[ArrayBufferData]] case, so I guess "transfer steps" could also throw an exception.
3. Set [[Detached]] to true and transfer the underlying MediaSource reference "during setup worker attachment communication".
   This approach might be tricky to extend to the scenario of unsetting from this media element attribute and setting on another media element.
4. Transfer the underlying MediaSource reference in the srcObject setter and transfer back when unset.

There is a slight difference in timing of options 3 and 4 due to awaiting stable state in the resource selection algorithm, and either of these timings could be applied to option 2.

[karlt]

Having the same kind of error path, as you describe, for multiple attachments of a single MediaSource, regardless of whether from Window or Worker context, makes sense.

If a MediaSourceHandle's [[Detached]] slot is true, then would this same path be taken?
(Consider transfer out of this Window context.)
If so, then explicitly saying here would be helpful.
But the decision here may depend on the choice of direction with the transfer-while-attached scenario.

I guess the alternative is to throw in the srcObject setter, but there is no precedent for that.

[wolenetz]

Thank you for your analysis and presentation of options. While I think each of those options are viable, I like options 1 and 2 best. They seem to be the simplest in concept and in implementation complexity. For option 1, the resulting MediaError.message could more readily indicate the reason for the error in that case. What are the timing considerations you mentioned might apply to option 2?

Note that initial prototype implementation in Chrome might just forgo some of the complexity involved in allowed sequential re-use of the same MediaSourceHandle (and add that later). The app can always create one or more extra MediaSource instances in the worker and get those handle(s) over to the main thread for readiness to use later as a workaround in the interim.

[karlt]

Consider the following two commands running under the same event (i.e. without a stable state between them):

video.srcObject = handle;
port.postMessage(null, [handle]);

With option 4, the postMessage() would fail and the video would keep playing.
With option 3, [[Detached]] is not set to true until the next stable state, and so the postMessage() would succeed and the video would error.

Option 2 may be spec'd by marking the handle as in use either in the srcObject setter for similar behavior to option 4 or in the "setup worker attachment communication" steps for similar behavior to option 3.

I haven't looked into the when exactly the handle would become unused if unset from srcObject, so there may be similar timing issues there for

video.srcObject = null; // previous value was handle
port.postMessage(null, [handle]);

Perhaps option 1 is immune to these subtleties.

[wolenetz]

@ #306 (comment) :

I haven't looked into the when exactly the handle would become unused if unset from srcObject, so there may be similar timing issues there for. [...]

Examining the current html5 spec, upon setting srcObject, synchronous execution of the media element load algorithm is done, which has a step that synchronously detaches MediaSource if one is the current assigned media provider object. So it seems that an implementation could readily know on the main/window thread whether or not a particular MediaSourceHandle object is currently assigned to any HTMLMediaElement as srcObject.

[karlt]

Ah, yes. Thanks for identifying that.
So, if choosing to block transfer while in use (i.e. not option 1), then for some consistency with main-thread MediaSource detach, I guess a postMessage() should succeed immediately after a handle is unset through srcObject.

[wolenetz]

@karlt, yes, I think I would prefer that.
I discussed this during today's Media Working Group Call, and also have shared this thread on the media WG mailing list to get wider input.
Previously, my preference was option 2 (throw exception if currently in use as srcObject value), however I have just now observed an issue with that option (perhaps this is what you were trying to point out, too):

video.srcObject = handle;
port.postMessage(null, [handle]);

While the handle is immediately set to be the media element's assigned media provider object, the synchronous portion of this setter (which runs the load algorithm, which then starts the resource selection algorithm) begins the actual attachment in an asynchronous set of steps once the next stable state has been reached. Therefore, the postMessage transfer mechanism, invoked synchronously before that stable state has been reached, does not know that handle is in a state preventing transfer. So neither option 3 or 4 can enable option 2 to prevent transfer before that stable state has been reached.

Does this mean option 2 is infeasible? Maybe, unless we update the srcObject setter spec for when a MediaSourceHandle is set as the assigned media provider object in the synchronous section of the steps following that assignment (for example, in the load algorithm). If we did that, then we could reliably know when the handle is assigned (or not) at any attempt to transfer it. (Note, unsetting the handle as assigned media provider object is already synchronously running MSE detachment as I noted earlier today.)

Overall, my preference is to lean towards giving application synchronous awareness of failure to transfer due to handle being in-use as srcObject already. I think this would help web authors avoid having to debug additional asynchronous error transitions that are already quite overloaded with various kinds of HTMLMediaElement and MSE errors.

[karlt]

I agree that, if choosing to block transfer while in use, then marking the handle as in use synchronously from the srcObject setter would produce the behavior with probably the least surprise.

@ #306 (comment) :
Examining the current html5 spec, upon setting srcObject, synchronous execution of the media element load algorithm is done, which has a step that synchronously detaches MediaSource if one is the current assigned media provider object.

FWIW I was assuming this detach was for the old/previous assigned media provider object. Perhaps that was intended, but that's not what the spec currently says. The srcObject setter "must set the element's assigned media provider object to the new value, and then invoke the element's media element load algorithm.", so the media element load algorithm detaches the new MediaSource object.

Still a synchronous release of the handle from in use when cleared from srcObject would be the least surprise behavior here.

[karlt]

The [[Detached]] internal slot provides single transfer for us, so your choice as to whether to have additional text here or not.

[wolenetz]

The intent is to prevent making more than one usable copy of a MediaSourceHandle for each MediaSource. Transfer semantics restrictions may not be sufficient, since basic serialization without transfer might be attempted.

I propose trying something like this (adding to my current rough prototype in Chrome):

• Change MediaSourceHandle to be just "Serializable" (https://html.spec.whatwg.org/multipage/structured-data.html#serializable-objects)
• For the serialization steps for MediaSourceHandle:
  • if the instance has already been serialized (e.g. some internal slot like [[AlreadySerialized]] is true), then throw exception.
  • Similarly, if forStorage is true, then throw exception.
  • If this instance is currently set as srcObject on any HTMLMediaElement, then throw exception. Note that this requires extension of HTMLMediaElement's srcObject setter synchronously update a handle internal slot, e.g., [[CurrentlyAttached]] to be true during setting srcObject to the handle (including resetting that slot to be false first for the previous srcObject if that object were also a handle.
  • Otherwise, serialize the internal state of the instance, set the instance's [[AlreadySerialized]] true. Note that this internal slot's value is reset false on the new instance resulting from deserialization.

Hopefully this adds clarity to the intent. I'll now try updating the Chrome prototype to do this and see how it goes and respond further here. @karlt and others, if you see any blockers or concerns, please report them here ASAP. Thanks!

[wolenetz]

e.g., [[CurrentlyAttached]] to be true during setting srcObject to the handle (including resetting that slot to be false first for the previous srcObject if that object were also a handle.

Note the HTMLMediaElement.srcObject setter doesn't have option to throw exception, but we need to synchronously let the handle know it's in the process of starting a load even before the asynchronous load operation starts. We also need to be careful to protect against the following situation:

// On main thread, handle is initially an unattached MediaSourceHandle
// just received from the worker that owns the underlying MediaSource.

media_element1.srcObject = handle;

// Would throw exception now, since handle is CurrentlyAttached:
// worker.postMessage(handle); 

// But this cannot throw exception. Rather, the mediasource attachment
// steps should later run the failure steps for media_element2:
media_element2.srcObject = handle;

// The following demonstrates that [[CurrentlyAttached]] cannot be a
// boolean since if it were, it would have been doubly set on the
// assignment of |handle| to media_element2, above:
media_element2.srcObject = null;
worker.postMessage(handle);  // If boolean, this would succeed, but shouldn't.

Based on that example, I propose changing [[CurrentlyAttached]] to be a counter
that is incremented when the handle is set to a srcObject on an HTMLMediaElement,
and decremented when the handle is explicitly unset from a srcObject on an
HTMLMediaElement. The attachment steps can then just fail if this value is not
precisely 1, and the serialization steps could throw exception if this value is not
precisely 0.

[karlt]

The intent is to prevent making more than one usable copy of a MediaSourceHandle for each MediaSource. Transfer semantics restrictions may not be sufficient, since basic serialization without transfer might be attempted.

I propose trying something like this (adding to my current rough prototype in Chrome):

* Change MediaSourceHandle to be just "Serializable" (https://html.spec.whatwg.org/multipage/structured-data.html#serializable-objects)

While I assume it is possible to specify a serialize-once behavior, I always thought of such a behavior where the object becomes unusable after message passing to fit better with the [Transferable] concept rather than [Serializable].

"Transferring is effectively recreating the object while sharing a reference to the underlying data and then detaching the object being transferred."

Serialization "allows them to be stored on disk and later restored".

Was there a benefit to making the object [Serializable] rather than [Transferable]?

[wolenetz]

Was there a benefit to making the object [Serializable] rather than [Transferable]?

Mostly this was due to the existing Chrome transferable infrastructure is clunky and might not do the required transfer steps in a completely interoperable order (from my brief reading).
Secondly, this simplifies postMessage to be just serialization, not making developers also put the object in a transfer list.
The conditional failure if "forStorage" part of the serialization steps allows prevention of usage for other purposes.

[wolenetz]

Regarding the specific postMessage exception:
If [[CurrentlyAttached]] > 0 or if [[AlreadySerialized]] is true, then a postMessage attempt including that handle should throw exception. If this condition is found first among other items being posted, then the exception should be a DataCloneError.

[wolenetz]

I've updated the chromium prototype change for this (circa patch set 28):

• handle copying disallowed. serialization of an instance can happen at most once (but can post a handle back and forth, assuming not currently attached).
• srcObject setter decrements previous handle (if any) attachment counter, and increments new handle (if any) attachment counter.
• loading steps in HTMLMediaElement do failure steps if loading via the handle and the attachment counter isn't precisely 1.
• Note: sequential use of the same handle in the same media element (or unset on one and set on another) fails on subsequent attempts, similarly to what might happen if attempting to load a revoked mediasource object URL.
• Note: the prototype doesn't yet disable dedicated worker MSE object URL creation, but that is the next part I'll add before getting it reviewed for landing in Chromium (along with tests).

I've tested the basic scenarios manually and it seems to be working.

[wolenetz]

@jernoble @karlt - I plan to proceed with getting the prototype in Chromium landed in parallel with updating this PR per the details as of #306 (comment)
Please raise any concerns you might have ASAP to prevent further delay on this highly-desired feature. Thanks!

[wolenetz]

@jernoble @karlt - I plan to proceed with getting the prototype in Chromium landed in parallel with updating this PR per the details as of #306 (comment)
Please raise any concerns you might have ASAP to prevent further delay on this highly-desired feature. Thanks!

Thanks to @sandersdan I am now aware of potential further complications:
It may not be interoperable to prevent postMessage(handle) from creating more than one handle, since this might actually be implicitly a broadcast on some implementations or on implementations where extensions might get a copy of messages. Achieving true move semantics is not possible. As a result, the window thread might eventually have multiple distinct handles referencing internally the same underlying worker-thread-owned MediaSource. It falls to the implementation to prevent simultaneous attachments (during the asynchronous media element load algorithm) from proceeding successfully, allowing at most 1 of them to attach successfully.
tldr: Keeping the [[IsAlreadySerialized]] proposed internal slot is not sufficient to prevent copies, but does help early detection of some of those scenarios. The media element load algorithm must fail if the same underlying MediaSource is already not 'closed'. The latter isn't a change to the spec.
Overall, Serializable is simpler ergonomically than Transferable, especially when there is potential for underlying broadcast or lack of true move semantics.

[karlt]

Browser implementations are free to use different approaches if they can produce the same observable behavior. Extensions can modify behavior in any way the browser lets them, if they are prepared to expect consequences, so browsers can't expect to maintain interoperable behavior in the presence of arbitrary extensions.

But what behavior would content see from passing a MediaSourceHandle to BroadcastChannel#postMessage(), if it were serializable?

While merely transferable and not serializable, MediaSourceHandle cannot be transferred through BroadcastChannel, which is consistent with transfer of MessagePort. "transferring does not make sense in multi-destination situations." MessagePort is similar to MediaSourceHandle in that it is a transferable object used for two-directional message-passing communication between threads and processes.

[sandersdan]

browsers can't expect to maintain interoperable behavior in the presence of arbitrary extensions.

As I understand it, all major browser implement the same behavior for message posting, which is broadcasting to all content script realms.

In WebCodecs we wanted to be able to have move semantics for VideoFrame but this detail made it impossible to implement in practice.

The situation with MediaSourceHandle is simpler because the resources it holds are less scarce, but I do want to recommend against inventing a new model for object ownership. If you don't follow the JS-like path (cloneable objects that are GC'd), there are surprising roadblocks.

But what behavior would content see from passing a MediaSourceHandle to BroadcastChannel#postMessage(), if it were serializable?

I would recommend that each receiver gets a valid handle, and that all valid handles are invalidated when one is attached.

While merely transferable and not serializable, MediaSourceHandle cannot be transferred through BroadcastChannel, which is consistent with transfer of MessagePort.

I don't think this is an option. Transferable isn't its own concept (or even an IDL attribute any more), it's a detail of the structured clone algorithm. If you can't serialize it, you can't transfer it either.

If there is a way to accomplish this combination, note that it would also prevent passing these handles through a TransferableStream.

[wolenetz]

I would recommend that each receiver gets a valid handle, and that all valid handles are invalidated when one is attached.

I agree. Further, for a particular handle instance, if it has ever been assigned to an HTMLMediaElement srcObject (even if the subsequent asynchronous load was interrupted and the underlying MediaSource hadn't ever been attached successfully), then that handle instance can never be successfully serialized again.

However, I think instantaneous serialization prevention of any handle instance in any realm with the same underlying MediaSource (such multiple instances can happen if extensions/broadcasts cause clones of the original single handle retrieved from MediaSource.getHandle()) upon any of those handles ever being assigned as srcObject to an HTMLMediaElement would risk security (could enable malicious high-resolution timer attacks).

So I think the following may be the best approach. If we find later that these restrictions can be relaxed, we can update the spec and implementations further:

• prevent a handle instance from being serialized if either it has already been serialized or if it has ever been assigned as srcObject to an HTMLMediaElement (regardless of whether or not that actually resulted in successful attachment+load)
• prevent the underlying MediaSource from being used (successfully loaded) more than once via srcObject on any HTMLMediaElement (similar to the existing object URL attachment mechanism immediately revoking the object URL on attachment start).

[karlt]

As I understand it, all major browser implement the same behavior for message posting, which is broadcasting to all content script realms.

That sounds very inefficient, especially for something large like a VideoFrame, at least unless there's shared memory access involved. But efficiency is much less an issue for transferring MediaSourceHandle.

In WebCodecs we wanted to be able to have move semantics for VideoFrame but this detail made it impossible to implement in practice.

I would guess that ReadableStream.tee() support requires a Serializable object. We don't need that for MediaSourceHandle.

I do want to recommend against inventing a new model for object ownership. If you don't follow the JS-like path (cloneable objects that are GC'd), there are surprising roadblocks.

There is no new object ownership model proposed here.
MediaSourceHandle can be GC'ed without any need to be clonable/Serializable.

While merely transferable and not serializable, MediaSourceHandle cannot be transferred through BroadcastChannel, which is consistent with transfer of MessagePort.

I don't think this is an option. Transferable isn't its own concept (or even an IDL attribute any more), it's a detail of the structured clone algorithm. If you can't serialize it, you can't transfer it either.

Transferable objects already exist without any need to be Serializable: MessagePort, ReadableStream, WritableStream, and TransformStream are all examples of objects that are Transferable, but not Serializable. If browsers can implement these objects, then I don't see any reason why a Transferable and not Serializable MediaSourceHandle would be a problem.

All these objects serve a similar purpose to MediaSourceHandle in that they provide potentially cross-realm communication with at most one associated other object. (tee() is an explicit extension of this involving additional objects.)

VideoFrame is very different. VideoFrames are data, not communication portals (end-points).

A sensible way to implement and even spec MediaSourceHandle would be that it hold a MessagePort created when MediaSource.getHandle() is called, to be used as [[port to worker]] on HTMLMediaElement.

If there is a way to accomplish this combination, note that it would also prevent passing these handles through a TransferableStream.

What do you mean by TransferableStream? If you can edit and link to a spec, that might be helpful, please?

We don't have any need to pass MediaSourceHandles through ReadableStream, etc.
MediaSourceHandles are not "created, processed, and consumed in an incremental fashion", they are not transferred in large numbers, and they do not need backpressure.