Mac, secondary monitor: Mouse jumps outside the Lix window. Allegro 5 bug, al_set_mouse_xy() miswarps

[Dominator101]

When attempting to play Lix on a secondary monitor on Mac there are some weird interactions with the mouse, hopefully I can properly convey what is happening.

The game window doesn't seem to 'capture' the mouse correctly. Instead the mouse seems to get locked outside the game window, and if you are to scroll the mouse over the window it instead snaps back to the seam between the monitors. In my case, I have my secondary monitor set up to the left of my laptop screen. If I have Lix on the left monitor and scroll the mouse over the window, the mouse snaps to the middle of the right edge of the screen, if that makes more sense.

If you give another program focus and hover the mouse over the window the Lix window, then click (to give Lix focus), the mouse will move around inside Lix correctly until you perform some action that 'refreshes' the view (ie, clicking a menu option, backing out of a menu, etc).

Totally get if this isn't a huge priority, or something that you can easily test (since I doubt you have a Mac lying around). If there's anything I can do to help troubleshoot or test, let me know.

OS details:
MacOS 10.14.5

[SimonN]

Thanks! It's rare to get reports for Mac, and I don't have one to test. Happy to hear that Lix works on Mac in general.

I haven't tested it with two screens yet. It's possible that there is something here even on Windows/Linux.

Are you running Lix windowed? I assume yes. Or software fullscreen, which is the default? To run it windowed, set the graphics options to windowed.

Does it help to enable/disable the option Fast Movement Frees Mouse?

Does it help to change the mouse speed between 10 and 20?

Will reply with more cultural background from the mouse-handling code within a day!

[Dominator101]

Are you running Lix windowed? I assume yes. Or software fullscreen, which is the default? To run it windowed, set the graphics options to windowed.

I generally run it windowed, I did try software fullscreen and it acted the same. Just tried hardware fullscreen and it...doesn't actually fullscreen apparently? Seems like a separate issue, but other than that same behavior as the other two with the mouse.

Does it help to enable/disable the option Fast Movement Frees Mouse?

Seems to act the same in both cases

Does it help to change the mouse speed between 10 and 20?

Doesn't seem to make a difference.

If it would help, I can see if I can record a quick video of what's happening. Not sure if it's something that I was able to properly convey via text.

[Dominator101]

Went ahead and made a recording anyway:

LixMouseIssue.mov

(If the embedded mov doesn't work, let me know, I can put it on youtube)

Also, looks like the issue is just any time you click, not necessarily something that causes a refresh (I guess I just always clicked on a different menu or something up until now).

[SimonN]

Thanks for the feedback and the video! This is a glaring, annoying bug. Your OS cursor warps to a completely wrong position outside the Lix window; according to your description, this warp target is near the seam of the two screens.

Expected instead: The OS cursor warps to the center of Lix and then hide; the warping shouldn't be visible, the OS cursor should merely appear to hide.

Make the Lix window partially cover both screens. Does the mouse warp elsewhere?

You've already explained that, when you put the Lix window entirely within the laptop monitor, the mouse warps elsewhere, and also not to the center of Lix.

Have the Lix window within one monitor. Disable the other monitor entirely. Does the mouse warp elsewhere?

Background for these questions: I've read the Allegro 5 source allegro5/src/macosx/qzmouse.m, function osx_set_mouse_xy(), it computes which screens intersect the Lix window, picks the first display from the returned array, and computes some target position from that.

From reading that A5 source, it doesn't look like an A5 bug and the math looks correct. But I'm not certain if their math is correct. That's why I'm suggesting these more experiments.

---

Lix's mouse trapping is custom-rolled: I want the OS mouse to exit the window with fast moves, but otherwise the OS mouse shall stick to the center of Lix and hide. It's normal that the Lix crosshair doesn't match the OS cursor; the Lix crosshair shall freeze whenever we right-click-scroll.

The source is src/hardware/mouse.d, handleTrappedMouse(). Also look around the remainder of mouse.d, there are hacks for similar problematic behavior on Linux and Windows. Allegro 5's mouse abstraction behaves slightly differently on the different OSes, and my mouse.d accumulated hacks around that different behavior until people were happy. I haven't touched it in years for Linux and Windows, but I would like to have it work on Mac, too.

[Dominator101]

I haven't tried having the window overlap monitors, but now that you mention it I think newer versions of Mac OS don't let you do that because of how they render each screen. I think the way it handles multiple screens is by making each one a different 'space' (I think that's the term) which is kind of like a separate desktop entirely, since it's a thing you can have even with just a single monitor (think like alt-tabbing between programs, except a totally different set of launched programs. You might have, say, chrome and notepad open on one and Excel open on another). I forget if Windows has this same concept or not, and I kind of assume at least some type of Linux does, so it might be something you're familiar with already.

But yeah, I think by default they render each monitor as one of those, so I don't think it will actually be shown on both at the same time, there will probably just be half of it on one and nothing on the other. I believe there's a way to disable that, so I'll probably do some playing around with that tomorrow.

I wonder if this is the root of the issue, though, it does make sense to some extent. Maybe what's happening is that it's trying to send the mouse to the center of the screen, but since the OS handles them as separate spaces it's trying to send to space A rather than B (where A is my laptop screen and B is the external monitor that Lix is on)? It would explain why it works on the primary monitor fine, and why every time the mouse jumps it goes to the center of the seam, though not why it stays on the seam rather than jumps to the center of space A...could be just be something with how the OS handles mouse movement between spaces or something, if it always sets it to the seam because that's where the mouse would be coming from?

[SimonN]

Please clone my exmousewarp example program; this is Allegro 5's own example for mouse warping that I converted from C to D. After cloning, enter its main directory, then build & run with $ dub.

Does the cursor warp to the center of the window when you press LMB/RMB? Or do you have similar warping mishaps as in Lix?

[Dominator101]

Tested the sample program, same behavior, on the primary monitor it works fine (assuming that something more up/left of the window center is considered the middle, looks like it's going to x:160, y:120). On my other monitor it looks like the x/y I'm getting in the console is x:320, y:240 (the window looks to be a different size on each monitor, though they've different resolutions so I'm not sure if they are in actuality or not).

Another note: I switched which display was my primary display, and the behavior switches with it. External monitor works fine and my laptop screen shows the warping issue (this time to the left side, as that's where the seam is now, so it is jumping to the seam and not just always the left).

As for testing with mac spaces disabled like I mentioned, I haven't gotten around to that yet as it apparently requires a logout and I'm still in work mode right now, so I'll try and test that at the end of the day. I tried some google searches for Allegro 5 mouse warping and mac, but nothing seemed to really come back, which seems a bit weird since I'd think someone would've come across this issue previously.

[SimonN]

Thanks a lot! (160, 120) is not expected, it should always be (320, 240) in the console. I'm getting close to filing a bug directly against Allegro 5.

Let me sleep over it 1-2 times; you too, take your time with testing after logging out.

[Dominator101]

So I did get around to testing with 'spaces' disabled (so the same desktop extends onto the second monitor) and I actually did still have issues with the secondary monitor. I made the window straddle the and did encounter issues most of the time, I think I did get it to slightly work on the right half of the window (that was on my laptop screen) once, but I also experience a lot of other general weirdness so not really sure.

I wonder if it's worth me trying to compile Allegro 5's example in it's native C, just for the heck of it. Might not make a difference but another data point shouldn't hurt.

If this does seem to be a general issue with Macs and external monitors, though, I'd be shocked that I can't find any reports of other people seeing it.

[Dominator101]

I did a bit of testing today with the Allegro c library examples, and I'm seeing the same behavior with the normal mouse warp example in their codebase.

I did notice this one note in a Readme about retina displays, though, not sure if it might be relevant or not?
https://github.com/liballeg/allegro5/blob/master/README_macosx.txt#L32

Unfortunately I haven't figured out if this is something I can change with the example programs or not (considering I'm kinda flying blind with little experience in these areas), might poke around a bit more and see if I can figure out if I can test with this option both on and off.

[SimonN]

Thanks! With your test against the unmodified Allegro 5 example in C, it looks even more like an Allegro issue. Consider to file a bug against Allegro 5 and link to this Lix bug #419. Allegro's IRC channel is #allegro on Freenode.

See also: Forestidia's thread on Lemmings Forums about this Lix bug #419.

My next idea would be to add debugging output to A5's osx_set_mouse_xy() and see what the coordinate mathematics do at each step. But that requires building Allegro 5 yourself. You already have a working C compiler and likely also CMake, you'd need to install its dependencies.

Re retina display fix: I don't know what Info.plist is; looks like Apple-specific file. Can't advise. But Retina display functionality doesn't sound entirely unrelated -- we're having trouble with screen coordinates.

[Dominator101]

I worked on getting Allegro 5 build this morning, that's how I built their C sample, so I can possibly play around with adding debugging to some of the samples.

As for Info.plist, yeah, that's a file that I believe is used by XCode as far as I'm aware, so it is Mac specific, but I'm not sure how it would be supported if you were building an app any other way other than XCode, so it makes me think there'd be another way to set it but haven't really found anything other than it being set in the Info file...It'd be nice to at least be able to test with it on/off to see if it even makes a difference. I think it requires a bit more setup to use this, though, as most of the samples don't have this, so I might need to hack one of the more complicated demos to see if I can play with this flag.

I just tested the warp sample with 0,0 instead of setting to the middle, and it looks like the position of the mouse on the seam does change, it jumps to the seam at approximately the top of the window.

On thing I did just think about, though, is that my external monitor wouldn't actually even be a retina display, it's just a normal monitor, only my laptop screen would be 'retina.' And since testing changing the primary monitor made the external work but not the laptop, not sure if the retina flag would even be the solution...that might just fix the issue with the values being half of the expected (I believe I did see some threads about people with retina that were returning half the expected value).

[Dominator101]

Just a quick update, I did try suggestion #7 from Forestidia in the forum thread, which i thought I had already tried but based on the results it's clear I hadn't.

I tried positioning the secondary monitor both above and below the primary, and the results were...strange. Rather than simply jumping to the seam, in both cases it acted more like the mouse was being 'pulled' down towards the bottom of the screen (even when the bottom wasn't the seam). Left and right would move the in-game cursor fine, but it would also move downwards even without any vertical movement. It was possible to 'fight' the pull to move the in-game cursor up, but overall it was just very strange behavior. Even stranger, it didn't seem like the real mouse was warped to a wrong location. Instead of the actual mouse showing up at the seam like the other tests, it instead seemed to actually stay overlapping the game window...if I alt-tabbed from the game, the real location was still overtop the game, so it seemed like the mouse was trapped correctly.

I may try to get a video of this later (though, wondering how well it will demonstrate the issue if you don't actually know where my mouse is actually trying to move), as well as try this with the Allegro sample directly (I've only tried this scenario with Lix itself so far).

[SimonN]

@Dominator101: Does your mouse still jump outside the Lix window when you use Allegro 5.2.8? That's the newest Allegro; it was released in June 2022.

[morien]

Reporting this mouse capture bug in Allegro 5.2.8 (still latest version available in homebrew), macOS 10.15.7

I see same behaviour as above. In windowed mode i lose focus all the time (mouse capture seems to be failing). In software fullscreen I get cursor jumping around if I venture near the left edge of the window. Same behaviour in primary and secondary monitor.

I can play non-scrolling levels fairly well, but scrolling makes things very challenging, as it's hard to access the left edge of the window without triggering these cursor issues

edit: i'm watching the video above again and the issues are much more pronounced than mine. in windowed mode, if i move the mouse slowly and don't press too much against the left edge of the window, i can play a scrolling level fairly easily. the trouble begins if i move the mouse too fast (capture can break, or cursor jumps elsewhere) or if i push against the left edge for a bit (capture breaks, cursor jumps right, then behaviour persists until i can touch the lower or right edge with the cursor, after which it calms down)

[SimonN]

Thanks for the update. Looks like it's still not fixed in Allegro 5.2.8 then.

Workaround: Scroll by holding the right mouse button anywhere on the screen, instead of scrolling by moving the mouse to the edge of the screen.