gamescope: Add Gamescope capture backend

[misyltoad]

Description

Add a Gamescope capture backend.

Motivation and Context

Providing support for streaming from Gamescope in both nested and session usecases.

A few options were considered here, such as implementing a screenshare
portal inside Gamescope, etc, but I decided to persue a Gamescope
capture backend directly for a few reasons:

1. Steam Deck support for OBS. Users will be able to share, stream and
   record their gameplay.

2. Many compositors do not support screensharing a single window, only
   the entire desktop. This allows a user to launch a game in Gamescope
   and immediately be able to share it.

3. This allows Gamescope to perform eg. color management, effects, have
   a Steam/performance overlay up for the user, but stream only the raw
   base-plane of the game. We can add a bunch of options to control what
   gets streamed on the Gamescope side. (Potentially also controlling that
   in OBS down the line with some custom metadata)

4. Doing a desktop portal for nested Gamescope (ie. run inside another
   compositor) would not be feasible.

5. Gamescope can run entirely headlessly, which can be useful in certain
   streaming scenarios.

How Has This Been Tested?

Tested that Gamescope capture works on my system and that the existing Portal capture code still works on my system.

Types of changes

• Bug fix (non-breaking change which fixes an issue) [for some prior commits]
• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

Stuff I need help with

Is pipewire: Move has_buffer workaround to non-dmabuf path correct? I am missing a lot of context here I think.

Would be cool if the people that know about that workaround would chime in.

[misyltoad]

The clang format stuff is failing even though I ran clang-format with:
git filter-branch --tree-filter 'git-clang-format; true' -- c71bbcf4d99039579e2c6996d55db691fb17a05c..HEAD which changed the formatting and seems to have worked. (Invokation taken from Mesa)

I am guessing it is actually due to a version difference? sigh

EDIT:

➜  obs-studio git:(gamescope-capture) clang-format --version
clang-format version 16.0.6

Even more confusing...

[RytoEX]

The clang format stuff is failing even though I ran clang-format with: git filter-branch --tree-filter 'git-clang-format; true' -- c71bbcf4d99039579e2c6996d55db691fb17a05c..HEAD which changed the formatting and seems to have worked.

I am guessing it is actually due to a version difference? sigh

clang-format has no guarantees of behavior remaining stable between any version, including patch releases. You must use exactly 16.0.5 for CI to pass.

[misyltoad]

16.0.5 was never packaged for Arch, only 16.0.6. Is there some container I should be using?

[RytoEX]

16.0.5 was never packaged for Arch, only 16.0.6. Is there some container I should be using?

You'll have to ask someone (or wait for someone) familiar with development on Arch. I work on Windows.

[misyltoad]

Okay, I got it. I had to pass a few extra arguments to git-clang-format based on the CI script and run-clang-format.

[kkartaltepe]

Just historically, we have explicitly avoided compositor specific capture methods and suggested compositors to agree to a standard implementation. To that end, that certain compositors do not implement window capture really doesnt seem like a problem we should workaround in OBS.

[misyltoad]

That would not solve the case of nested + only streaming the base plane without color management, etc applied.

Gamescope also works entirely headlessly, so this can be used for other forms of streaming also.

[kkartaltepe]

That would not solve the case of nested

Has this been raised with the xdg portal folks? If its important enough for us id imagine we want to solve it for everyone not just obs, and the only way that happens right now is in xdg portal.

Gamescope also works entirely headlessly, so this can be used for other forms of streaming also.

We also avoid supporting headless configurations, fwiw.

[misyltoad]

This doesn't really link in to xdg-portal imo. There is no access control or anything related to the session/host compositor going on here. If Gamescope isn't the user's session desktop session, or OBS is not ran inside Gamescope, then I don't think we can really make this work with that.

I do want to support xdg-portal screensharing/capture in Gamescope when it's used as a session at some point, but I would say solving the nested/headless without overlays case is pretty separate to that.

[tytan652]

This is a personal opinion:

There is no hole punched in the Flatpak sandbox required to allow PipeWire Window/Screen Capture to work since it uses only xdg-desktop-portals.

You PR is clearly requiring one, which is the wrong direction imo.

[GeorgesStavracas]

Hey @Joshua-Ashton, have you considered raising the shortcomings of xdg-desktop-portal's screencasting APIs to portals developers at github.com/flatpak/xdg-desktop-portal/ ?

Looking at the last commit message, I think it's possible to implement all except one of the points in a portal backend. From what I understand, as per what @kkartaltepe said, headless is probably not something that the OBS community supports now.

A potential gamescope portal backend could even proxy portal calls to other backends if gamescope is not running, if so you wish.

[misyltoad]

OBS Flatpak already hole-punches the pipewire socket because it needs to record the mic + desktop sound.

I am guessing the goal is maybe to remove that at some point and have some portal permission system for that?

Hey @Joshua-Ashton, have you considered raising the shortcomings of xdg-desktop-portal's screencasting APIs to portals developers at github.com/flatpak/xdg-desktop-portal/ ?

Looking at the last commit message, I think it's possible to implement all except one of the points in a portal backend. From what I understand, as per what @kkartaltepe said, headless is probably not something that the OBS community supports now.

I have considered it, but it seemed like it wouldn't work ie. if you are running Gamescope inside of another session (KDE/Gnome, etc) with their own desktop portals.

Gamescope itself could also be run inside of a Flatpak potentially someday, so not really sure what the path is.

Happy to make an issue there to start discussion (maybe there's already an existing one about mic/speaker capture to feed from?)

To me it's sounding like the problem with this being hole-punched is the exact same as solving the hole-punching for mic/speaker capture. It's just another pipewire node after all.

[GeorgesStavracas]

I am guessing the goal is maybe to remove that at some point and have some portal permission system for that?

Yes, that's the ultimate goal. Static permissions were a necessary step to bootstrap the technology, but it's pretty clear now that most of them should become portal-based dynamic permissions.

I have considered it, but it seemed like it wouldn't work ie. if you are running Gamescope inside of another session (KDE/Gnome, etc) with their own desktop portals.

I see. The next stable release of xdg-desktop-portals, which I'm going to release today or tomorrow, includes a complete rewrite of how portal backends are picked, and possibilitates installing and layering backends through a config file.

On Steam Deck, you would be able to configure xdg-desktop-portal such that it would use the (theoretical) gamescope backend even on a KDE session. The (theoretical) gamescope backend could, then, proxy D-Bus calls to the KDE backend through a private D-Bus interface.

Interested to hear if you think this could cover the Steam Deck use case.

Happy to make an issue there to start discussion (maybe there's already an existing one about mic/speaker capture to feed from?)

Please do so! Thanks 🙂

Gamescope itself could also be run inside of a Flatpak potentially someday, so not really sure what the path is.

That's a neat idea, running a nested compositor through Flatpak is an interesting challenge. There were some small experiments with GNOME Shell doing that years ago, but it didn't lead anywhere. Maybe we could talk about this separatedly, elsewhere.

[misyltoad]

Interested to hear if you think this could cover the Steam Deck use case.

This isn't just about Deck. This is for users on desktop too, and eventually using big picture mode in Steam as well (whenever that transitions to be Gamescope + lease based somewhere down the line).

Can't really have a flatpak having its own portal in the way and stuff like that.

Back onto the "this is just really about exposing arbitrary pipewire nodes". I do think that this really is actually just that.

The real question is how are we going to solve this in the general case? I am sure we cannot expect every app doing audio/video to do all of it's negotiations thru dbus/desktop portal. Something here reallllllly has to be transparent to the application somehow or we are going to be going through hundreds of apps and fixing them.

Maybe it is a simple as a coarse "portal to get the pipewire socket fd". That's a pretty big hammer, but, hear me out:

One thing we have been wanting for a while is pipewire namespaces -- where applications using pipewire can only see a subset of nodes. One main usecase for this is eg. game streaming using Steam In-Home Streaming and giving the apps a virtual speaker/virtual mic and making sure they only see that; also VR, and making sure apps launched thru SteamVR will only see the HMD's audio devices.

I wonder if there is some common ground here in exposing apps a pipewire fd that is namespaced to have only certain nodes (ie. for this it would be all input + output + video) thru a portal. Desktop environment's portals would probably just be like "yeah whatever" for output/video nodes, prompt for input nodes, etc.

Maybe this is already possible in some way, I am not super sure. The people I have asked so far have told me it is not possible, otherwise I'd be doing it right now in a bunch of places :D

That's a neat idea, running a nested compositor through Flatpak is an interesting challenge. There were some small experiments with GNOME Shell doing that years ago, but it didn't lead anywhere. Maybe we could talk about this separatedly, elsewhere.

Gamescope already works in Flatpak already I think? https://github.com/flathub/com.valvesoftware.Steam.Utility.gamescope

[dimtpap]

One thing we have been wanting for a while is pipewire namespaces -- where applications using pipewire can only see a subset of nodes.

I wonder if there is some common ground here in exposing apps a pipewire fd that is namespaced to have only certain nodes

I think this is how the screencast portal works
It connects to the existing PipeWire remote on the system but sets the permissions on its client to only see certain nodes: https://github.com/flatpak/xdg-desktop-portal/blob/5744dc33747da7cd07189fddc802ef9eac8ec463/src/screen-cast.c#L675-L685
Then it steals the connection fd of the client and hands it over to the application requesting OpenPipeWireRemote:
https://github.com/flatpak/xdg-desktop-portal/blob/5744dc33747da7cd07189fddc802ef9eac8ec463/src/screen-cast.c#L1002
Finally, the application that got that fd then can only see the appropriate nodes.

For new nodes, you can dynamically update the permissions of that restricted client from another, more privileged client, to let it access them

[misyltoad]

Thanks for pointing this out, that seems really interesting to experiment with for what I was interested in doing. Was not aware it could work like this. Will feedback on that soon.

I'll make an xdg-desktop-portal issue soon about the pw hole-punching -- but currently thinking that the PR should be fine as-is (or close-to as-is), and will automatically get solved when we solve that.

[TheEvilSkeleton]

Gamescope already works in Flatpak already I think? https://github.com/flathub/com.valvesoftware.Steam.Utility.gamescope

Yep (although new ID is org.freedesktop.Platform.VulkanLayer.gamescope)

[norihiro]

Just a nitpick. The commit title should have a prefix linux-pipewire: instead of pipewire: and gamescope:.

[GeorgesStavracas]

Taking a fresh look at this PR again.

The following commits can be removed:

• c58fe1d
• c1cece6

The following commits can land independently of this pull request (please open a new pull request so we can land them):

• e6e7abb
• af8b0d2

The rest still needs proper review and scrutiny

[Quackdoc]

I've only taken a glance, but Is this actually gamescope specific? It seems like this is just a generic pipewire capture while specifying gamescope as a node name? This seems like, given minimal modification, it should work with any pre-existing pipewire video source if you change the node name?

for instance, given the example gst pipeline gst-launch-1.0 videotestsrc ! videoconvert ! pipewiresink stream-properties="p,node.description=test_out,node.name=test_out,media.role=camera,media.class=Video/Source,node.autoconnect=true" mode=provide sync=false

would it not be possible, given minimal modification, to have it support this by replacing node name gamescope with test_out and have it work?

[misyltoad]

Yes, it would work.

[Quackdoc]

Perhaps it might be better to put a simple "Generic Pipewire Source" or whatever better name then that then? if one of the reservations was;

To that end, that certain compositors do not implement window capture really doesnt seem like a problem we should workaround in OBS.

it seems like a generic pipewire capture might be better. This would likely also have the immediate benefit of also adding better libcamera support since from a quick search on git, doesn't seem like OBS currently supports.

we can get the list of devices using commands like pw-dump | jq '.[] | select(.info.props["media.class"] == "Video/Source") | .info.props."node.name" ' under which gamescope does show up. better would be using node.description for showing as the name to theuser since it gives the device names, however it appears gamescope doesn't set that. (it probably should IMO).

I for one would love support for this since I do use gstpipelines to make windows that OBS can record. and I haven't been able to do this on sway due to a lack of window capture on sway. (pw-v4l2 hasn't worked for me).

As for xdg-portals, perhaps a portal could exist to provide access to these nodes mentioned above? I will say it would be really nice if this wasn't something compositors would have to deal with. We are seeing more compositors that either have lesser support for xdg-portals, and even some compositors that don't have their own backends. Im not to sure how the xdg portals people would like to handle it, but using arbitrary pipewire nodes would really be great

[tytan652]

it seems like a generic pipewire capture might be better.

No, like it was said no new feature forcing a punched hole to be kept is a good design. An alternative is in WIP (link bellow).

This would likely also have the immediate benefit of also adding better libcamera support since from a quick search on git, doesn't seem like OBS currently supports.

OBS interact with PipeWire though portals, which will interact with libcamera devices through the Camera portal which is in WIP.

• Camera portal (feat. PipeWire) #9771

As for xdg-portals, perhaps a portal could exist to provide access to these nodes mentioned above?

• App to App Media Sharing flatpak/xdg-desktop-portal#1141

This is WIP and I'm working on a proof-of-concept. I think that the Gamescope usecase is covered by it.

We are seeing more compositors that either have lesser support for xdg-portals, and even some compositors that don't have their own backends. Im not to sure how the xdg portals people would like to handle it, but using arbitrary pipewire nodes would really be great

xdg-desktop-portal is the new common API for desktop environment, if those compositors don't want to support/implement them. It's unfortunately their problem. OBS Studio will not support them with a custom implementation.

Edit: And I repeat Gamescope has a usecase that is outside of the scope of most compositor, which I try to cover with this new portal.

[Aeonitis]

Hey, everyone, does this mean that Gamescope don't need to do anything to resolve the issue of OBS Gamescope capture since they already have pipewire functionality and required access features as previously stated already?

I guess I shouldn't bother them with my questions on the Gamescope repo yesterday, should I close that?

I understand that you added it as a future milestone. Hope you can educate me on this if possible.

[kkartaltepe]

If you want to see action on the xdk-desktop-portal you should probably voice your support for tytan's proposal and your use cases on the xdg-desktop-portal discussion linked. This PR will not be merged as is, if it was not clear..

[Conan-Kudo]

The only portal backend I'm aware of that doesn't support window capture is the wlr one. All the other ones do support window capture. But with the new ext-image-copy-capture protocol, this should be fixable in xdg-desktop-portal-wlr rather than working around it here.