Dock previews not available for apps running in another Space

[stevehoek]

Many similar Dock preview apps run into this problem. Apparently apps like DockView and DockMate use a Private API to get the window preview for running windows in other Spaces (not the current Space).

Could you look into this? Having only some windows show their Preview takes away from the mission of a Dock preview app.

[hasansultan92]

+1 AltTab is able to solve this issue and I would want to see this functioning in DockDoor too

[ejbills]

Do you have any info on this private API?

[stevehoek]

Do you have any info on this private API?

Unfortunately no. There was another thread in r/macapps from someone who is building a dock preview app and he informed me that DockView and DockMate use it and that he refused to. If it can't show me all apps, I'll stick with the buggy DockView that is slightly less buggy than DockMate

[ejbills]

Ok

[hasansultan92]

Can we perhaps reach out to the team at AltTab? I have been using that for years and they have it figured out. Seems to me that they have the GPL 3.0 license which my understanding is that we are allowed to copy and distribute but not change the codebase without permission. Seems to me that we can go over their codebase to understand how they sorted it potentially.

I could potentially look at this later in the week if this is still not taken up by someone

[stevehoek]

AltTab is open source: https://github.com/lwouis/alt-tab-macos

[hasansultan92]

Right, so would it be that bad of an idea if we inspect how they did it since we don't have access to the aforementioned private APIs?

[ShlomoCode]

Do you have any info on this private API?

from https://alt-tab-macos.netlify.app/contributing:

OS APIs are quite limited for the kind of low-level, system-wide app AltTab is. This means often we just don’t have an API to do something. For instance, there is no API to ask the OS “how many Spaces does the user have?” or “Can you focus the window on Space 2?”. There are however, retro-engineered private APIs which you can call. These are not documented at all, not guaranteed to be there in future macOS releases, and prevent us from releasing AltTab on the Mac AppStore. We have tried my best to document the ones we are using, as well as ones we investigated in the past.

[stevehoek]

FWIW, another tool I've been watching just implemented multiple-spaces support, as well as full screen windows and iOS/iPad-app on Apple Silicon windows. Does not show the "screen recording" menu bar icon so I don't think it is using the private API.

https://wins.cool/

[ejbills]

Interesting, I'm curious how they are achieving that. I'm already half-using the new screencapturekit API, but in my testing it was limited to the current desktop space. I will download the developer beta and will try it out again.

[ShlomoCode]

It just means that it uses the cache at the expense of freshness, while alt-tab performs a background update. DockDoor now uses a cache with a minute expiration (which is a lot! in fact, too much) and no background refresh, so it's unlikely to have the "screen recording" icon like alt-tab-macos
From my testing, the icon appears when many screenshots are taken in a row. If I uncache the screenshots in the current DockDoor code, and quickly hover back and forth over an icon in the Dock, the purple icon appears:

So it's not related to the type of API but to the technique of using it.

[ejbills]

Wasn't talking about the lack of the screen recording notification, more about how the 'wins' app has support for multiple desktop spaces. I suspect once I migrate to the screen capture kit method of capturing window images, the screen recording notification will be mitigated as it is a new system. I will do some more testing.

[ShlomoCode]

I suspect once I migrate to the screen capture kit method of capturing window images, the screen recording notification will be mitigated as it is a new system.

I don't see a reason for it to be like this, after all it comes to warn that an application is watching the screen for a long time. What does it matter through which API.

[hasansultan92]

Hi everyone, I am circling back on this issue and picking up development. I have some findings and I am looking to get customer/audience/user feedback on how we should realistically be implementing this, knowing the constraints we know now.

First of all, huge thanks to AltTab, I have learned quite a bit in the past couple of days through them. I have used this application for the past couple of years, and have adored this application.

Challenge:

MacOS seems to not allow the native present/released/public API to allow detecting all "windows" that the user might have open. If one calls the native API, it spits out everything open in the current user space. A space is defined as the current desktop where all your widgets and applications are open. One may have Safari open in space X but also another Safari instance open in a focus mode in space Y. Depending on which space you call the API from, the API will only return windows open in that space. There seems to be a good reason why they are doing it, but there also seems to be a way around - so good news there.

If you have 10 spaces, including monitors, you will only see the windows from space x wherever DockDoor might be residing at the time of calling that API.

Solutions:

Three solutions are coming to mind, AltTab issue 447 has had a thorough conversation on this and I do not deem it necessary to bring up all their points.

1. Usage of a daemon at Mac launch 😀

This is mentioned in the issue I have linked above. Im not very well versed on daemons but it seems that Mac OS allows daemons to be spawned at the time of turning the computer on? it also seems that Mac OS allows the daemon to capture the information and then relay that information back to the "master" application - in this case DockDoor.

The concept is, the user turns on the computer, Mac OS gets in memory, boots, the user logs in, applications launch and cause chaos, the daemon walks along and launches with the chaos, daemon records who was open and who launched in which space, daemon keeps track of them, chaos dies down and applications stabilize, DockDoor launches, DockDoor pulls information from daemon, we kill daemon and proceed with our day.

• Pros

  1. We have access to all the windows at launch

• Cons

  1. Mac OS launch is kind of a chaos
  2. Everything launches in the same space, pretty useless to us if this continues in the future MacOS releases
  3. What do we do after launch? How do we keep track of what is open still and what is not once the user physically moves all the open windows to their desired spaces? We can potentially use Solution 2

2. Move everything in one space & back 😊

Another solution I am not very well versed with, but came to my mind anyways.

DockDoor launches, DockDoor uses private APIs to first call all the windows to one space, then once it "discovers" them, we attempt to place them back to their respective spaces they came from.

• Pros

  1. User cannot find a window thats open? phuft, we run the function, move everything around and we find the "missing" window.
  2. Always able to find our windows

• Cons

  1. Presently no officially supported APIs, heavy use of private APIs
  2. Might trigger the "application is recording" light
  3. Might flash the screen for a brief second like a screenshot, similar to BUG: When hovering over dock icon, screen flashes white for a split second #78
  4. Might not be able to place them back, no way to track who came from where as of my knowledge today, more experimentation required
  5. Already not a huge fan of the OS messing up where my windows are and now DockDoor is potentially allowed to mess them up. My personal experience with Mac OS seems to be that the OS likes to move windows from space X to space Y any chance it gets -Apple if you are reading this, please please fix this-

3. Concept of discovery 👍🏼

This is one of those solutions I am well versed with right now. AltTab is also using this along with some other neat features to do the job.

Two scenarios:

1. Suppose DockDoor launches at the startup of Mac:

Ideal scenario: ..., User logs in, Application chaos at launch, DockDoor joins the chaos, Applications all launch in space X, DockDoor knows which windows are opened in space X, the user moves the windows around, DockDoor has already discovered and learned about the windows beforehand -process ID is the same - so it can find them again.

New windows launched after startup and while DockDoor was running will all be tracked from here onwards

As of today's scenario: Not sure, have not gotten the chance to test it

1. Suppose DockDoor launches today even though my Mac has been running for days:

DockDoor has no idea who was open before its launch, we cannot get information on these windows because even if we use a private API to find out about the existence of application windows open, AXUIElement and company are not able to give us the window preview, ways to close the window, and minimize the window, etc. Essentially we use a private API to find out what windows and applications are open, we piece up that information, but AXUIElement will tell you "Oh I am sorry, this does not exist" even if it does.

The way around is, that DockDoor launches, DockDoor uses private APIs to get all windows open, filter out the ones that aren't real, and present the ones AXUIElement allows for us to get previews of and other functionality, etc. To allow DockDoor to "discover" other windows, the user just goes to the window and makes it active/uses it and DockDoor will now have information about this window since AXUIElement cannot deny that discovery. By nature, AXUIElement is not allowed to return information about windows from other spaces. Once DockDoor can discover this window with the help of the user, the user can continue to do the previous functionality.

• Pros

  1. Easy to integrate with our current functionality - time to market is the fastest.
  2. Some applications seem to be able to return their windows through AXUIElement even though they are in another space. I had two 3 Safari windows open that DockDoor had not "seen" yet but AXUIElement was able to return information about those windows
  3. Fairly simple
  4. Can integrate with system notifications and potentially cache information on windows to figure out real-time when the user is moving them around and make future computations cheaper

• Cons

  1. Usage of this method involves a combination of official and unofficial APIs
  2. Inconvenient to roam around in the spaces and click here and there to find all the windows that are already "open"

Conclusion

Above is a summary of what I have been up to, let me know your thoughts and please bring yours to the table too. My initial goal was to try my best to not remove existing functionality and reuse existing functions and structures where possible. We can look into overhauling the system, but we should think of the cost of doing that - time to market, time in development, time in testing, viability for the future.

React with the respective heading emojis so I can gain some idea of where the audience is leaning towards

[hasansultan92]

FWIW, another tool I've been watching just implemented multiple-spaces support, as well as full screen windows and iOS/iPad-app on Apple Silicon windows. Does not show the "screen recording" menu bar icon so I don't think it is using the private API.

https://wins.cool/

@stevehoek this link does not seem to be working, can you link it again. If I recall this application correctly, this is a system level application and not a high level application like DockDoor. I am not sure how the developer was able to make an application like this on Mac OS that is baked into the system settings, if someone knows, I would love to learn on this topic too.

[stevehoek]

I think I messed the URL insert up in my comment... the site is https://wins.cool

They say they are "system level" but I don't know what they mean by that. They integrate into System Preferences using a .prefpane extension

I've been using it for about 2 weeks and found it works well, discovers all windows after they have been user activated once, can handle all window types (iOS apps, sandboxed apps, full-screen apps, normal apps). However, I notice that after awhile the previews stop and i have to toggle Wins off and on again to get the preview back (and activate each window again)

AltTab also has the issue sometimes of needing to activate a window before it shows up in AltTab... mainly full screen windows in my experience (Parsec remote desktop comes to mind).

The Daemon idea is a unique one afaik, and might be useful to do some prototyping of that.

I'd be willing to test anything you want!

[ShlomoCode]

@hasansultan92 I don't like the idea of ​​inventing our own wheel, after others have spent years of work and discussion to find the best way.
For example see https://github.com/lwouis/alt-tab-macos/blob/master/src/logic/Applications.swift#L31-L46

[hasansultan92]

@ShlomoCode they are doing a lot more than those lines that you have sent. They have their own types and notification handlers handling a lot more. As I've mentioned above, they are doing something along the lines of Solution 3 and some more. If we go their route fully, we have to change a lot from what we have already established. My last run of AltTab v6.64 on Mac OS Sonoma does the discovery concept that I have mentioned above.

[ShlomoCode]

DockDoor's architecture is pretty non-existent right now. It's actually a good idea to rewrite and also organize, based on OOP, like alt-tab. I don't see a disadvantage in a big refactor, but an advantage.

[ejbills]

It's actually a good idea to rewrite and also organize, based on OOP, like alt-tab. I don't see a disadvantage in a big refactor, but an advantage.

What major benefits would be get from doing this, and how could it be applied? The singleton hover window is for the sliding animations. There is also a singleton for managing the global WindowSwitcher/FullScreenPreview states (which hooks into the mouse tracking utility or key bind listener utility which have to be global observers). Curious how the workflow would work. Also, I refactored a lot of the code for window management today, and I think it is a lot cleaner now.

But, I do think if we continue to modify the frontend window display there will grow to be a large amount of tightly coupled code. It’s important to weigh the immediate challenges of a large refactor against the long-term benefits. I can't imagine the hover window singletons will need much changing, beyond the addition of the full size preview I added today. That is just frontend wise, though.

[ShlomoCode]

The OOP approach means that we have a "type" that contains all its operations and information as methods/properties, like this
We just pass WindowInfo directly to a bunch of generic functions in WindowUtil

[ejbills]

The OOP approach means that we have a "type" that contains all its operations and information as methods/properties, like this We just pass WindowInfo directly to a bunch of generic functions in WindowUtil

I will put in some effort to move the window manipulation funcs from WindowUtil into the WindoInfo definition. Besides that, I am not sure there will be much benefit.

[ShlomoCode]

There is also a lot of abstraction that can be done, for example all the messing around AXUIElement. But his license is different from ours :(

[ejbills]

But his license is different from ours :(

Not anymore, just updated.

[ShlomoCode]

it is great! Many parts of the work (in fact everything except what is related to the Dock) are actually shared, and we can save a lot of work by using snippets of its code