App to App Media Sharing

[tytan652]

Related to Goal: Peer-to-peer media sharing

Usecases

Case n°1

Applications like Inochi 2D want to be able to share a stream (maybe even various) to other applications (e.g. OBS Studio) alike to what Spout2 and Syphon allows to do on respectively Windows and macOS with almost zero-latency, so mostly "PipeWire + DMA-BUF" for Linux.

We can also point out Gamescope that is a Wayland compositor that want to enable sharing in nested scenarios where a xdp implementation is not a solution (obsproject/obs-studio#9607).

Case n°2

Others applications want to be able to provide a virtual camera (e.g. OBS Studio) to be used as a camera in other applications.
Some of them already does by using v4l2-loopback which require kernel space API to be used.

As an example, OBS Studio plans to move features to PipeWire (obsproject/obs-studio#7998), but the actual proposed implementation requires an access to the host PipeWire.

Case n°3

Also some applications might want to act as virtual microphone (device to send their own audio stream) alike to one of Easy Effects features but without requiring direct access to the host PipeWire.

Idea of a solution

Note for the following:

• Shown sequence diagram are not strictly sequence diagram
• I didn't deep dive into how PipeWire nodes should be handled, I might do it at the same time that I work on a proof of concept of case n°1

For case n°1

For this use-case, we need to enable applications to provide screen cast sources (virtual) to the ScreenCast portal of the xdp implementation.
So the xdp stack will have to connect to the application point to point D-Bus to make it a screen cast sources provider.

Note: Name in italic are just temporary name

It is also imaginable to simplify the per-application implementation by providing a library that provide most of the basic of the implementation.

Thought about audio on those Screencast

Also related to Goal: Window screencast with audio.

Since the application provide its own video stream, the application could also provide a audio one paired with the latter.

This could improve the Gamescope sub-usecase, and maybe also allow to some power users of OBS Studio to "stream" an OBS session to another locally without having to rely to solution like NDI or DeckLink if implemented in OBS. Yes, chained OBS Studio setup like that do exist.

For case n°2

I didn't give enough thought about it, and like I said earlier I need to dive a little more on how PipeWire works with Portals.

For this one, we need to enable applications to give a video stream to xdp to make it a camera available through the Camera portal.

There is no expectation that the virtual camera can do more than an actual camera, so format-wise it should be like a real camera.

Reminder: Name in italic are just temporary name

For case n°3

This cases is alike to the case n°2 but relies on Goal: Audio portal/#1142.

But unlike case n°2, the naming "virtual microphone" is just used as a figure of speech.

[Mikenux]

From a UX point of view.

For case 1, I think there could be two possible experiences:

1. Direct sharing of the streams from the sharer app. The user is presented with an "Open with" type dialog box with apps that support receiving streams from sharers. Permission to share the streams is granted when the user has selected an app and clicked “Open”. In the dialog box, we can use more appropriate phrasing and words than “Open (With)”.
2. Grant access to the streams from the requesting app. The user is presented with the source selector from the app with which to share the streams. If this is secure, sharers can expose their streams without asking the user. If not, apps capable of sharing their streams report that they are able to do so. Then, when the selector is called from the requester, two possibilities, depending on security: (1) send a signal to open compatible apps so that they share their streams, (2) list open compatible apps and send the signal when the app is selected. In short, the ability for an app to provide its content to other apps is done from the selector, since it is the portal.

There may be an option to share or not share audio from a stream.

For case 2, either we display a dialog asking to add the camera (confirming its creation) and it is directly available for other apps in a selector, or it can be the same as in the second experience of case 1 but for a virtual camera.

[tytan652]

Direct sharing of the streams from the sharer app. The user is presented with an "Open with" type dialog box with apps that support receiving streams from sharers.

1. We can't know in advance which app can/will do what since those have access to all portals by design, so no "Open with".
2. The plan for the new portal is to be a "provider" to other portal (ScreenCast, Camera, Audio) without adding over-complexity to them, only those portals UI/UX will be used including the Access one.

If this is secure, sharers can expose their streams without asking the user.

This is not how permission and the ScreenCast portal works:

• if the permission is unset, the user must be asked if the app can share
• if it's not a session restoration the user must be asked with a selector.

There may be an option to share or not share audio from a stream.

Out of scope, those of thought for the quoted goals, this is to talk about in another discussion about adding audio to the ScreenCast portal.

[Mikenux]

In the second experience, I talk about implicitly granting access to streams from the relevant portals (called "source selector" because we select sources with them). I mean we want to avoid "Allow %s to provide screencast sources".

[tytan652]

I mean we want to avoid "Allow %s to provide screencast sources".

The application must explicitly have permission separately from and before the ScreenCast dialog, there is no point to avoid it.
If the permission is not given, the portal implementation must not be aware of the provider.

[Mikenux]

This is wording that should ideally be avoided, not to not grant permission. Here, it is not common, but it is about displaying in the screencast UI a list of open applications having self-declared (or using any other valid hints) to be able to give their content. Clicking on a listed application means that you are implicitly requesting and giving permission for that application to provide its content. This is, I believe, more or less the same for the OpenURI portal: this dialog is displayed from application A, but you are implicitly giving permission to application B to open a link or file that the application A "has".

Also, I think if the streams are pipewire sources, then they can be presented directly in the screencast UI. I don't think you're granting permission to a portal to access things, because the portal is the one granting access.

[tytan652]

I began by saying that the whole permission block in case n°1 is there because I thought allowing explicitly the sharing app to share would a good idea, the permission looks valid only if asked while the sharing is declaring itself to xdp and not later in the xdp impl ScreenCast selection. If various consider it unnecessary with discussions and agreement, it will be removed.

In my opinion, you are just asking for the removal of this block.

but it is about displaying in the screencast UI a list of open applications having self-declared (or using any other valid hints) to be able to give their content

This portal will not enforce UI/UX guidelines to xdp impls, if they want to show an apps list and then a sources list, or just a sources list. It's up to their design guidelines.
Xdp impls will just be made aware of the sharing apps (if allowed) by xdp, and it will be up to them to show them and their sources with their own UI/UX.

Clicking on a listed application means that you are implicitly requesting and giving permission for that application to provide its content.

"implicit" is different from "explicit", for me the former has no value as a permission. And like I said before the xdp impl have directly access to the sources list.

Also, I think if the streams are pipewire sources, then they can be presented directly in the screencast UI.

Nope, in the ScreenCast portal, PipeWire streams can be created after the source happen to be requested by the xdp impl, they don't have to exist beforehand.

[Mikenux]

Could you describe case 3 further? Is it, for example, an app that applies voice effects and then shares the resulting audio stream?

[tytan652]

This will used as a sort of changelog if I happen to update my illustrations.

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v1 and v2:

• Replaced application interface names
• Removed the "maybe ?"
• Various wording changes
• Changed how how the xdp request application sources

[tytan652]

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v2 and v3:

• Application ScreenCast Session design is overhauled
  • When submitted as a provider, no session is created but the D-Bus name is watched and the token is stored for later
    • If the D-Bus disappear, the token is destroyed
    • Potential future addition: Checking if the D-Bus name has the needed interface
  • Sources list is given by the application ScreenCast interface (not the session)
    • If the token is wrong, the provider is ignored
  • The Applicatrion ScreenCast Session is created if a application stream happens to be selected

[tytan652]

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v3 and v4:

• Move to a point to point D-Bus connection design for the providing app
  • I want to thank Christian Hergert for releasing Collaborating on Builder where I discovered this
  • Just in case, it's Unix Domain Sockets as a transport on the D-Bus spec
  • Removes all to token complexity
  • Removes all the same App ID or D-Bus name stuff since it's now mostly file descriptors

NOTE: I'm working on a POC with point to point to see if it's viable. If it's not, I'll go back to v3 I guess.

[tytan652]

PR for case n°1: #1554