More Linux Work

[Erotemic]

This is work on this branch that will hopefully help improve Linux support. I'm trying to get this to work on my system.

Eventually it would be nice to have proper python packaging, as right now it's just a set of scripts which is unwieldy. It should be entirely possible to set this up through pip and have it be pip installable. See my other projects for how I usually do this.

[Erotemic]

Thanks for taking a look. After failing to press hotkeys to advance my splits on SM64 16 star last night, I'm more motivated than ever to get this working.

That's something I'd like to look into (I'm guessing using setup_tools or similar?).

The first step is to get this working on Linux. If I was to set this up, I would likely use setuptools because that's what I'm familiar with. I've recently started work on a library called xcookie, which I'm building to help me template new Python projects or update existing ones to have a proper package structure. There' still a lot of me-specific hard coded stuff in there, but the idea of:

• have a folder with an __init__.py that is your package. That is the one source of truth for where __version__ lives. The setup.py script parses it.
• enumerate requirements in a folder of independent requirements with a base requirements.txt pointing at those base requirements. Platform specific requirements are dealt with using conditional statements, not separate files (e.g. jaraco.windows>=3.9.1;platform_system=="Windows").
• write a setup.py (and a pyproject.toml). The setup.py parses the __init__.py for the version and the requirements.txt for the packages. Any optional packages are dealt with using "extras".

I have a lot of experience with Python package setup and management. I'm not opposed to using poetry (although I've never used it directly myself), but I'm also very willing to take on this task of restructuring this as a proper package. But I think getting the minimum viable product is the first step.

WRT to pyenv: I have a bash script that helps me automate the setup of pyenv. Pyenv already does a good job itself, but this handles details like ensuring system packages are installed (assuming debian / ubuntu), configuring python optimizations, etc... the script itself is documented in a way that it gives examples of how to run it. It does depend on my bash utilities script, which has even better documentation. (I need to figure out a good way of distributing bash libraries).

No work has been done yet to support multi-monitor on Linux. So this is a TODO :)

I'm not even 100% sure that's the problem. But its worth looking into.

This error is actually an "old" one. You have to install from git git+https://github.com/boppreh/keyboard.git#egg=keyboard, they haven't yet published an official update on PyPi.

I'll check that out, if the authors don't want to maintain the updated version, we can always fork it to remove the dependency on a git repo. I have quite a bit of experience publishing pypi packages, so I could do that if it would help. I do think eventually moving to pynput, where the system is entirely permissionless might ultimately be the best longer term move, but I may be missing something that keyboard can do that pynput can't, and also the quicker fix sounds like the right scope for a "get Linux working PR".

That one's just broken unrelated to Linux. This feature gets no love XD

I can likely fix it. I'm fairly familiar with opencv.

I've made a small list of things I know are still missing to have feature parity with Windows: #33 (comment)

If you'd like a more direct channel for communication, don't hesitate to contact me on Discord: Avasam#0612

Will do, I'm bossofthegame#9288 on discord. Using github with async conversations would be best for now, but I'll check that if you want to ping me.

[Avasam]

For now I'll just add a warning with instructions if the user isn't in the input group.

What I most need help with atm is making sure Capture Devices work (since I can't make my c920 work through my VM). As well as supporting multi-monitor setups.

We also need to figure out how to find the size of the shadow around windows (like Finder) so that I can properly offset the targeted region, like we already do like on Windows. If this isn't something PyWinCtl can help with, then adding the functionality to the library will be beneficial to reduce code complexity on our side. But we need to figure it out first.

Feel free to fix the Align Region feature separately. It's very low on my priority list and it's probably just a format issue.

[Erotemic]

For now I'll just add a warning with instructions if the user isn't in the input group.

It turns out that keyboard doesn't even let you install it if you don't have the right group permissions setup. It's kind of annoying. I think we don't need to check for that explicitly in this program given that they do it, and a bit of documentation will go a long way here. Ultimately, it would be best to replace keyboard anyway, so I don't think it's worth investing time there.

What I most need help with atm is making sure Capture Devices work (since I can't make my c920 work through my VM). As well as supporting multi-monitor setups.

If you need help testing something I help with that if you give instructions. I have a native Linux setup. I can disable my second monitor if we need to verify something works with/without multi-monitor. I'd hope that it wouldn't be that big of an issue though.

We also need to figure out how to find the size of the shadow around windows (like Finder) so that I can properly offset the targeted region, like we already do like on Windows. If this isn't something PyWinCtl can help with, then adding the functionality to the library will be beneficial to reduce code complexity on our side. But we need to figure it out first.

Sounds like a computer vision problem. I'll give it a think.

Feel free to fix the Align Region feature separately. It's very low on my priority list and it's probably just a format issue.

For whatever reason, I can't seem to reproduce it.

I did find out that PyWinCtl really doesn't like whatever window the Mario64 PC port uses to display itself though. I'm guessing it might be doing some low level thing to display itself, which could be causing issues. It does work normally for other windows.

[Avasam]

It turns out that keyboard doesn't even let you install it if you don't have the right group permissions setup. It's kind of annoying.

I've noticed. It runs dumpkeys during install, which really doesn't feel intended. It's been messing up the CI (GitHub actions) because I can't set the active user's groups and running pip as root breaks Pyright linting...

I wouldn't mind moving away from keyboard. Even on Windows you may have noticed some very specific code to listen for numpads + PyAutoGui to send them. And it still sometimes break on non-english locales... Also Toufool#155
I've been eyeing pynput for a long time now. And our hotkeys module could benefit getting rewritten from scratch. It's just that ensuring support for modifiers, numpads, and multi OSes is a big ask.

Sounds like a computer vision problem

It's less computer vision and more just finding the right XWindow property and doing a simple substraction :P Alternatively I think it's possible with Xlib to obtain the render of the window itself. That may simplify a lot (and increase performance). For inspiration, they seem to do it. It may still render the border shadows though.

---

Something I'll absolutely need help with, since I cannot get either my webcam nor OBS' virtual cam plugin working on linux (so I can't even test this):

If you have a webcam, or any capture device, that you know works, can you see if AutoSplit picks it up? And if it doesn't work... you can have fun with OpenCV and/or V4L2, cuz cannot work on that at all.
But that's also non-blocker, that feature is not mandatory.

[Avasam]

I've opened an issue on keyboard's side concerning dumpkeys on install: boppreh/keyboard#563

[Erotemic]

I followed instructions on setting up permissons:

  sudo usermod -a -G tty,input "$USER"

  # TODO: this might be necessary, but I'd like to justify it before I script it.
  # Can we work around it?
  sudo chmod +0666 /dev/uinput

  # Writing these udev rules seems necessary, but need to happen as sudo
  sudo_writeto /etc/udev/rules.d/50-uinput.rules 'KERNEL=="uinput", TAG+="uaccess""' 
  sudo_writeto /etc/udev/rules.d/12-input.rules 'SUBSYSTEM=="input", MODE="0666" GROUP="plugdev"' 
  sudo_appendto /etc/udev/rules.d/12-input.rules 'SUBSYSTEM=="misc", MODE="0666" GROUP="plugdev"' 
  sudo_appendto /etc/udev/rules.d/12-input.rules 'SUBSYSTEM=="tty", MODE="0666" GROUP="plugdev"' 

  echo "
  The user has been ensured to be in the tty and input group. You may need to
  restart or logout and login before this change can take effect.
  "
  loginctl terminate-user "$USER"

I'm still getting:

ERROR: Failed to read device '/dev/input/event11'. You must be in the 'input' group to access global events. Use 'sudo usermod -a -G input USERNAME' to add user to the required group.

But it seems to run, so I'm ignoring it.

Yes, AutoSplit does pick up my webcam:

It seems to list them multiple times. I can't take a screenshot while interacting with the dropdown, but the items are:

• Logitech BRI [V4L2]
• Logitech BRI (occupied)
• Logitech BRI [V4L2]
• Logitech BRI (occupied)

For capture methods I have XDisplay and Video Capture Device listed.

The entire program does seem very sluggish though.

Other things I noticed:

• The code does not gracefully handle the case where the window being tracked is closed. Probably an issue not needed to be considered in this PR.

Also getting this error when setting hotkeys:

Traceback (most recent call last):

  File "/home/joncrall/code/Auto-Split/src/hotkeys.py", line 264, in read_and_set_hotkey
    hotkey_name = preselected_hotkey_name if preselected_hotkey_name else __read_hotkey()

  File "/home/joncrall/code/Auto-Split/src/hotkeys.py", line 213, in __read_hotkey
    return __get_hotkey_name(names)

  File "/home/joncrall/code/Auto-Split/src/hotkeys.py", line 188, in __get_hotkey_name
    clean_names = sorted(keyboard.get_hotkey_name(names).split("+"), key=sorting_key)

  File "/home/joncrall/code/Auto-Split/src/hotkeys.py", line 184, in sorting_key
    return not keyboard.is_modifier(keyboard.key_to_scan_codes(key)[0])

  File "/home/joncrall/.pyenv/versions/3.10.5/envs/pyenv3.10.5/lib/python3.10/site-packages/keyboard/__init__.py", line 427, in key_to_scan_codes
    normalized = normalize_name(key)

  File "/home/joncrall/.pyenv/versions/3.10.5/envs/pyenv3.10.5/lib/python3.10/site-packages/keyboard/_canonical_names.py", line 1239, in normalize_name
    raise ValueError('Can only normalize non-empty string names. Unexpected '+ repr(name))

ValueError: Can only normalize non-empty string names. Unexpected ''

closing and then redoing it seemed to be fine though. But if i reset focus it happens again. Also closing settings after this seems to prevent settings from reopening.

But working through it I can get the hotkeys set.

The Select Window seems to be working now when using M64Py emulator instead of the sm64-port binary.

I got it to successfully recognize the start image, but unfortunately it seems m64py eats any hotkeys and nothing gets picked up by livesplit-one. :( But that's not an issue for this repo I don't think.

[Avasam]

I get the error message as well. But it seems we can safely ignore it (it's just printed to the console anyway, not actually raised) boppreh/keyboard#312 (comment)

The entire program does seem very sluggish though.

The entire code about capturing AND comparing images still runs on the main thread. So the entire program starts running at the same framerate you can capture frames... (yeah not great ik) (Unless the capture method implements ThreadedCaptureMethod, but that's more of a quick fix) So that probably doesn't help.

Doing an actual screencast and only extracting the required frames may also be more efficient than taking screenshots. Btw that's why I had to drop gnome-screenshot support. Since GNOME flashes the screen every, single, time. We'll HAVE to use screencasting for Wayland support.

I also need to find a way to capture only the targeted window, rather than the entire desktop + cropping. It's one of the reasons BitBlt (as well as WGC) is so performant on Windows.

The code does not gracefully handle the case where the window being tracked is closed. Probably an issue not needed to be considered in this PR.

Maybe not in this PR, but that's something that will need to be supported. Recovering the window of a game after it was closed (either Save & Quit or it crashed) is an important feature that is currently supported on Windows. If I can replicate I'll create an issue so it's tracked.

Do you know what you're doing exactly to get empty hotkeys? Either way I'll tackle it as part of this existing issue: Toufool#155

I got it to successfully recognize the start image, but unfortunately it seems m64py eats any hotkeys and nothing gets picked up by livesplit-one. :( But that's not an issue for this repo I don't think.

On Windows I'd say check if you're running the app "as admin" (because an app as admin won't let other apps listen to its keystrokes). I'd bet there's something similar on Linux where you can allow another app to listen. Unless m64py is suppressing the keys.

[Avasam]

I'll close this PR as stale. But feel free to update it and re-open it.