Window Management

[morsssss]

Request for Mozilla Position on an Emerging Web Specification

• Specification Title: Window Management (previously: Multi-Screen Window Placement)
• Specification or proposal URL:
  • explainer
  • specification
• Caniuse.com URL (optional):
• Bugzilla URL (optional):
• Mozillians who can provide input (optional):

Other information

incremental improvements to existing screen information and window placement APIs, allowing web applications to be intentional about the experiences they offer to users of multi-screen devices.

• Support requests to show elements fullscreen on a specific screen
  • Extend Element.requestFullscreen() for specific screen requests
• Support requests to place web app windows on a specific screen
  • Extend Window.open() and moveTo()/moveBy() for cross-screen coordinates
• Provide requisite information to achieve the goals above
  • Add Screen.isExtended to expose the presence of extended screen areas
  • Add Screen.change, an event fired when Screen attributes change
  • Add Window.getScreens() to request additional permission-gated screen info
  • Add Screens and ScreenAdvanced interfaces for additional screen info
  • Standardize common Screen.availLeft and Screen.availTop attributes
  • Add Permission API support for a new window-placement entry

[dbratell]

Heads up: This now has a "request to ship" in Chromium.

[morsssss]

Thanks for pinging this up! If someone has time, we would still like Mozilla's position on this.

[annevk]

I appreciate the detailed explainer here. The Privacy & Security section cites a TAG principle around device enumeration. That actually seems concerning to me as exactly these kind of bits are valuable information to segment and exploit end users. I would be much more comfortable with the website doing some kind of request to present its information and then after the user grants it that power it getting more detailed information about the screen(s).

I also miss considerations here around platform independence, a core principle of the web. A lot of work has been done to overcome early X11-centric design decisions and looking at the list of proposed changes in OP this adds them right back in.

[michaelwasserman]

Thank you for providing this feedback, @annevk!

1. Does your concern about device enumeration pertain solely to the Screen.isExtended bit? (or also with Window.getScreenDetails() and the overall API design?)

   1a. The Screen.isExtended bit is an important boolean signal for the site to know whether it is worth asking the user for additional screen details at all. That said, we want to allow different browsers to adopt different permission models. So, the spec should probably state what it looks like if the browser doesn't want to reveal the plurality of screens via Screen.isExtended. Perhaps it could be left undefined or always set to true, requiring or encouraging explicit async requests to equivalent information via Window.getScreenDetails().

   1b. More generally, we explored alternative API shapes that avoided or limited device enumeration. Browser-based screen pickers seemed cumbersome for users, offering little utility over manually dragging windows between displays, and ultimately exposed the associated Window.screen object for the chosen screen.

2. Can you clarify your meaning about the incorporation of X11-centric design decisions, and past work done to overcome that?

   2a. This API mainly exposes per-screen details in multi-screen environments, which also allows introspection of the virtual screen arrangement. That, and the ability to specify a target screen for fullscreen requests, seems pertinent to all modern windowing environments. This API seems like a reasonable incremental step toward enabling broadly applicable multi-screen capabilities, by offering the requisite information for opinionated window placement.

   2b. While the API also extends the web platform's existing coordinate-based placement paradigms to multiple screens, the aforementioned multi-screen information could also be used in future APIs designed specifically for more modern and ergonomic multi-screen windowing requests. We documented some related API compatibility risks (e.g. offering lower-fidelity placement in Wayland-based desktop environments) and forward-looking explorations. Future non-fullscreen window placement APIs could formally expose the coordinate-based placement capabilities of the host device, take target screen arguments along with and optional per-screen coordinates and/or target window states.

I look forward to discussing your feedback further!
It would have been even better to discuss design at an earlier date.
Thanks,
Mike

[morsssss]

Like Mike says - thanks for offering this feedback!

As you can imagine, quite a bit of work has happened since I posted this request last June. We've traveled far down the road to launch. But this feedback is quite helpful. We'd like to see what we can do to address it, while minimizing potential breakages for any early adopters out there.

In other words, if you have time to get to Mike's questions as soon as possible - say, this week - this will help us address your concerns and improve the API for everyone.

Thanks,
Ben

[martinthomson]

Hi @morsssss, @michaelwasserman,

I've discussed with this @annevk and in addition to Anne's reservations about the X11-style model (on which I have no opinion), we are concerned that the additional passive exposure of fingerprinting information has not been given enough weight in your analysis.

The work includes a few pieces:

• Support requests to show elements fullscreen on a specific screen
  • Extend Element.requestFullscreen() for specific screen requests
• Support requests to place web app windows on a specific screen
  • Extend Window.open() and moveTo()/moveBy() for cross-screen coordinates

So far, this looks fine, but obviously you can't place things on a screen without knowing how to identify that screen. How sites obtain this information, as proposed, is where this falls off the rails. Users with multiple screens are rare and exposing that information passively carries a risk. We'd prefer if this were based on some sort of user action.

You appear to have considered an option that we'd prefer:

Show multi-screen users user-agent controls to move windows between screens.

This seems like an avenue worth exploring more. The browser will know if a window/tab moves to a new window, so it doesn't need user agent-native controls for moving windows; this could rely on observing when windows move by any means.

This could be coupled with an event that fires when a user moves a window to another screen (maybe after a small delay). Information about that screen can then be learned by the site. Until a screen is known to a site, the list of screens only includes the current one. Sites can encourage users to manually place a window, once, in order to learn about additional screens.

The advantage of this is that there is no increase to the passive fingerprinting surface and no additional prompting. The permission grant is implicit and tied to an action that a user understands. That makes this more like <input type=file> than getUserMedia... in a good way.

You claim some drawbacks of this approach that I'd like to address:

• Again, the resulting window.screen information is likely still available.

I'm not seeing evidence that this is the case. If I am a multi-screen user and I constrain the use of a site to one screen, what information are we leaking to that site such that they might learn about the existence of other screens? If there is something, maybe we should be looking to fix that.

• This could be as cumbersome for users as dragging windows to target screens.

This is in our view a reasonable compromise in exchange for more effectively protecting multi-screen user privacy. The one-off cost for users and websites is modest.

• This does not allow web applications to suggest optimal initial placements.

Once the screens are known to the site, they can suggest optimal placements. I don't imagine that multi-screen optimizing sites are hyper casual in nature such that this will be a huge problem (counter-examples welcome, of course).

• This does not allow web applications to save or apply user preferences.

I don't see how this is a problem. Once the site knows about the additional screens, we can remember that they know and expose this information later. So save away.

• This may conflict with OS-specific paradigms for similar window controls.

This is a UA problem and one that seems manageable by using the OS controls if they exist.

Though the current shape of the API is what we might consider harmful, we could be more positive with this sort of change.

[michaelwasserman]

Thanks for the quick reply and for brainstorming here; privacy has been an important consideration in our design process all along. If I understand correctly, you suggest that a site could gain persistent access to each screen where the user explicitly places its windows.

I think that the currently proposed API shape allows the implementation you're proposing. Could you please follow my outline below and check that conclusion?

1. Sites call window.getScreenDetails() which the browser can resolve (or reject) by:
   a. Optionally prompting the user. A prompt could include a picker to expose a selected set of screens. Perhaps you'd prefer not to prompt, and always resolve the promise.
   b. The resolved ScreenDetails interface may only expose a limited set of screens initially, perhaps just the current screen, just the screens previously 'visited' by the site, or the above picker-selected screens.
2. As the site's windows move to new screens, the browser can expose corresponding ScreenDetailed objects in the ScreenDetails.screens array and fire the screenschanged and currentscreenchanged events.
   a. A browser may do this when the user invokes browser-specific window placement controls, or anytime the user places the window on a new display by any means.

There would be many implementation-specific details to resolve with this approach, but it seems feasible and fairly well-aligned with the existing API design. While Chrome wouldn't take this approach (at least for now), I wrote up some initial considerations and concerns for this approach that I would be happy to share separately.

Let me try to address some of the other pieces you mentioned:

we are concerned that the additional passive exposure of fingerprinting information has not been given enough weight in your analysis. ...
Users with multiple screens are rare and exposing that information passively carries a risk. We'd prefer if this were based on some sort of user action.

Can you extrapolate on this? The W3C definition of "passive fingerprinting" does not apply here. JS API usage is detectable on the client and can only be used for “active fingerprinting”. Fingerprinting concerns and mitigations are documented in the API’s Privacy Considerations, which cites requirements of explicit user permission and activation. Please also consider my concrete suggestion to mitigate the active fingerprinting surface of isExtended above.

the current shape of the API is what we might consider harmful

Hopefully the API shape's compatibility with your suggested implementation approach and picker-based approaches might encourage you to reconsider the perceived harm of this API.

Your point-by-point analysis of my screen picker drawbacks is helpful. My naive screen picker brainstorming only considered prompting users on each multi-screen placement, not granting persistent screen access for future placements of any window.

Your combined suggestions, to gate screen information and placement capabilities on manual placement and/or screen picking steps, seem like viable implementation approaches compatible with the current API design. At the same time, the API design also allows browsers to offer greater user convenience in a single permission prompt (or none at all), so web applications could correctly arrange multi-screen content without necessitating manual window dragging or screen-picking steps by the user.

If I am a multi-screen user and I constrain the use of a site to one screen, what information are we leaking to that site such that they might learn about the existence of other screens?

My point was that by moving the window from one screen to another by any means, the site may have observed the window.screen information for each of these screens.

Thanks again to both of you, @martinthomson and @annevk! If you're interested, we can supplement this discussion with a one-off W3C Second Screen CG or WG meeting. (+CC @anssiko)
Mike

[annevk]

A problem with allowing implementation freedom when it comes to UX is that websites will nonetheless build around a particular pattern. So even if the specification allows for "Website tells the user to use multiple screens so it can make use of them.", websites might not implement that particular pattern leaving users of user agents that implement that pattern in the dark. We've seen this problem manifest before with other APIs.

[martinthomson]

Yeah, I don't think that UX freedoms apply if the result is that sites need to act very differently. As @annevk says, we standardize so that sites don't have to do that.

[michaelwasserman]

Standardizing an access model that requires users to move windows between displays as a prerequisite for enumerating screens would necessitate an exceptionally poor user experience, and it would not offer substantial privacy improvements over permission-gated enumeration.

That said, the proposed API shape is flexible enough to allow experimentation, and I agree that sites should not need to act very differently. So, a user agent itself may need to engage with users directly, in order to implement novel access models and still support the underlying use cases.

Our separate envisioned patterns to enable multi-screen content placement capabilities seem fundamentally compatible, and I still believe that both can be achieved with the proposed API shape. Thanks for your continued insights.
Mike

[michaelwasserman]

I'd welcome discussion at the Second Screen WG/CG meeting in May: w3c/secondscreen-wg#4

[anssiko]

I've put this topic on the meeting agenda w3c/secondscreen-wg#4

@annevk @martinthomson thanks for sharing your perspectives in this issue. You're welcome to participate the upcoming meeting as guests and/or (continue) provide your feedback in an async manner in this issue -- whatever fits your work mode, we'll cater for that. All feedback is welcome and will be considered.

[pawansingh00]

FYI: This already seems to be implemented and shipped as part of Chrome 100 -
https://developer.chrome.com/blog/new-in-chrome-100/#multi-screen-window-placement

Tried in my system and seems to be working perfectly in Chrome, If we can get it in Firefox too, will be super useful.

const x = await window.getScreenDetails();
// returns
// {
//    currentScreen: {...}
//    oncurrentscreenchange: null
//    onscreenschange: null
//    screens: [{...}, {...}]
// }

The below request popup comes up when we do this -- window.getScreenDetails

But accessing this - window.screen.isExtended doesn't require any specific permission.

[michaelwasserman]

@martinthomson: I appreciate the recent continued discussion at the Second Screen vF2F; I hope we'll continue these discussions before/during/after the upcoming TPAC. Please reach out at any time; thanks!

[michaelwasserman]

I welcome the opportunity to continue discussing this topic at TPAC! If anyone is interested, we can add a Second Screen WG/CG agenda topic, or expand the scope of my tentative breakout session.

[michaelwasserman]

I invite your consideration of w3c/window-management#100 and w3c/window-management#130, which clarify security and privacy considerations and suggest potential mitigations aligned with feedback from this issue and #636.

Discussion at the upcoming Second Screen WG/CG - 2023 Q1 virtual meeting #7, in this issue, or elsewhere is greatly appreciated. Thank you!