Implement itch.io as a new game service

[neuromancer]

Reference implementations:

• https://github.com/itchio/itch
• https://github.com/itchio/butler

[tkashkin]

It doesn't look like they have required functionality in their API. Also OAuth applications can only request profile (https://itch.io/docs/api/oauth#the-authorization-step/scopes).

[neuromancer]

Can you just use butler to download and update games? (I know it is not the best solution..)

[neuromancer]

An interesting feature to consider from the itch.io client is the use of a sandbox.

[impiaaa]

It looks like the official client uses a different API, used and sort-of-documented here. Specific endpoints of interest might be /login, /profile/owned-keys, /games/<gameid>/uploads, /games/<gameid>/download-sessions, and /profile/game-sessions/summaries/<gameid>. Additionally, it seems like Playnite was able to get started by contacting the itch devs directly.

[hagabaka]

Hi, I have made some progress on this. https://github.com/hagabaka/GameHub/tree/itch

Currently I'm able to login with an API key, and fetch a list of games, although I haven't figured out why they're not showing up in the GameHub UI (it's able to print a list of their names in console).

This is much more complicated than other data sources, because butler daemon uses a flavor of JSON-RPC 2.0 over TCP which doesn't seem to be supported by jsonrpc-glib, so I had to make my own implementation. The documentation for this API is currently here: http://docs.itch.ovh/butlerd/master

The code is missing a lot of error checking, and I'm probably doing a lot of things the wrong way. I'd appreciate any help and directions.

[tkashkin]

@hagabaka thanks. I have fetched your branch to https://github.com/tkashkin/GameHub/tree/itch.

I have added some error checks and improved butler detection. By default butler is not in $PATH if it was installed with itch desktop app. This case should be handled correctly now.

I can't actually test this since I don't have anything on itch.io. I assume Fetch.ProfileOwnedKeys should only return bought items.

Run GameHub with --debug --verbose arguments to see logs.

[hagabaka]

Thanks. I've merged the commits and will keep working on it. Fetch.ProfileOwnedKeys returns free games as well as purchased games (). It should be printing a list of game names in the console even if you haven't bought anything...
I think it returns all the games listed in https://itch.io/my-purchases . My list definitely shows a lot more games than I actively bought though.

The games are actually displayed in the "all games" tab now, but not the Itch.io games tab. Any idea why? Now the app shows the list of itch games correctly.

[hagabaka]

I've added the functionality to list, install, and run itch.io games. Here are some of the remaining issues:

• Installing a game uses the first version of game offered by butler, which I think is the latest version for your platform. There is no functionality to install an alternative version, and installing games with no Linux support will just fail. Butler does provide the information needed to implement this, but I'm not sure how to implement the UI for it. We just need to display a dialog with a few choices, and return the choice the user selected. InstallDialog seems interesting, but it takes a list of Runnable.Installer which I think need to have URLs, and butler doesn't provide them.

• There is no indication of "downloading" or "running". Butler provides these through JSON-RPC notifications. However they are not automatically associated with tasks. To properly handle them, it recommends opening dedicated connections for longer running tasks like installations and launches, which will only receive notifications for the tasks they start.

• Updating game information on demand isn't implemented yet, so the information is only loaded from butler when GameHub starts.

• Uninstalling a game isn't implemented yet.

• Running with compatibility tool isn't implemented. Currently we use Butler's Launch request to run games, which doesn't support compatibility tools. I'm not sure what the data source needs to do to support this actually.

[tkashkin]

Yes, Runnable.Installer should be implemented for itch.io games. It may require some changes but should be possible.

Maybe Runnable.Installer should be abstracted more and split into

• Installer that is something more abstract that does nothing and only has metadata like id, name, size.
  ItchGame.Installer should inherit Installer and interact with butler.
• DownloadableInstaller which will be same as Installer is now.
  Anything that inherits current Installer should inherit DownloadableInstaller instead.

---

Downloader and Download/PausableDownload are abstract classes. Probably there should be itch-specific Download implementation which will manage butler connection, listen to events and update download status.

---

Running with compatibility tool isn't implemented.

Is it possible to get game's executable path from butler? It should be possible to use compatibility layers in that case.

[tkashkin]

@hagabaka I have refactored Runnable.Installer in 38f7c2b.
Now there are Installer, FileInstaller and DownloadableInstaller base abstract classes.

Now GameHub should load installed free itch games in addition to owned games.

Also some GameHub features are implemented for itch games.

[hagabaka]

@tkashkin Thanks. To update status during game install and launch, I think we need to split ButlerDaemon into two classes, one which starts the daemon and gets the address and secret from output, and one (ButlerConnection) which maintains a connection to the daemon and sends/receives messages. This way we can use a main ButlerConnection for general tasks, and create a temporary ButlerConnection when starting an installation or launch, and handle notifications in it which will be related to that task. This is basically how itch.io recommends using the daemon API. I know you simplified the ButlerConnection away by combining it with ButlerDaemon before, but that makes it hard to use multiple connections to the daemon. What do you think?

[aaronfranke]

Summoning @fasterthanlime, he may be able to help, though his status say's he's on break.

[fasterthanlime]

@hagabaka Using separate connections for tasks like launching/installing/etc. is a good idea indeed! Closing the connection will "cancel" these operations. This is the concept of "conversations" as implemented in https://github.com/itchio/node-butlerd

Here's an example usage of node-butlerd:

https://github.com/itchio/itch/blob/308582de070376fb017a4f2917e42a6d8116cdeb/src/main/reactors/updater.ts#L73-L109

Having a separate connection lets you route logging (the Log message) elsewhere, cancel separately, not mix up server requests (that should show dialogs, etc.). It's low-tech, but it works.

Like @aaronfranke said, I'm on semi-break, but happy to review butlerd client implementation before it goes live (I will be afk Sep 1-9 though).

[hagabaka]

I've added status update during game download. There are a couple ugly details:

• To get a "downloading" Game.Status, it needs to be constructed with a Downloader.Download. And this class requires remote, local, and local_tmp files/urls to be set. The actual download is performed by butler daemon, which doesn't provide all this information, so I just set them all to "/dev/null", and that seems to cause no problems. I think Game.Status should only require a Downloader.DownloadStatus, which contains the download progress and speed.
• Speeking of which, Downloader.DownloadStatus requires the currently downloaded size and total size. Butler daemon's Progress notifications only give a progress percentage. So I resorted to using the totalSize in the TaskStarted notification to calculate the downloaded size. If for any reason we fail to get that notification, it would display "0/0 0%", even if we do have the current percentage. Maybe the status should only require a progress, and allow you to display any other text information. For example, butler provides an "eta" field too.

[tkashkin]

@hagabaka I have refactored download status so it doesn't need workarounds like this.
There's now ItchDownloader which implements Downloader and has signals to update UI. That allows to integrate downloads into UI.
Installer is also implemented and now InstallDialog is used to select version to install.

[hagabaka]

@tkashkin Cool! That works great for me.

I added handling of ShellLaunch, HTMLLaunch, and URLLaunch server requests. These are used when butler tries to open an HTML game. We're just using xdg-open right now, and some games don't work because they need to be served in a local HTTP server, but I don't think this would be a high priority issue for most people. I implemented this just so that GameHub isn't stuck in "running" when I click on such games, because the Launch call waits for our response to return.

It seems to me that the only main missing feature is running with compatibility layer. Butler doesn't have an API to tell us the name of the game executable. We could try to guess it, but there would be risk of running "uninstall.exe" etc. Butler has code to handle this (through dash), but it doesn't tell us the result, and it filters by platform by default. I created an issue to ask for the butler API to provide this information, which will make launching with compatibility much easier to implement. There's also an issue requesting for itch/butler itself to support compatibility tools.

[hagabaka]

@tkashkin What other missing features or implements do you think we need to implement before merging this?

One possibility is updating games. Butler has an CheckUpdates API, which returns a list of Uploads that can be installed with similar calls as installing games. However GameHub doesn't seem to have UI for updating games. Should it automatically check for and install any updates before running a game?

I implemented game updates before launching. It works, but the UI isn't very intuitive as it looks the same as freshly installing a game. Is there a way to make the InstallDialog display "Install update"?

[spinningD20]

Looking forward to using this! What else needs to be done before we merge it in? If you need me to test anything, I can give it a shot.

Thanks for the work on this, and for gamehub in general, I really like the project.

[tkashkin]

@spinningD20 testing would be nice.
If it works and it's stable I can probably find some free time and merge it.
Some kind of game updates UI for this and #323 would be nice, but meanwhile updates on game launch are fine.

[spinningD20]

[ERROR]  [GLib-GIO] Settings schema 'com.github.tkashkin.gamehub.paths' does not contain a key named 'itch-home'
Trace/breakpoint trap (core dumped)

There might be a previous settings file somewhere, but the above is what I'm getting when I do ninja in the build directory after doing a meson in the base directory, and attempt to run the compiled binary.

I'm on ArchLinux fwiw.

EDIT: I removed .cache/com.github.tkashkin.gamehub files and tried it again, got the same result.

[tkashkin]

@spinningD20
You need to build package from this branch or run sudo ninja install to install updated settings schema.
Alternatively you can try to set schema directory before running compiled binary:

export GSETTINGS_SCHEMA_DIR="$APPDIR/usr/share/glib-2.0/schemas/:$GSETTINGS_SCHEMA_DIR"

[spinningD20]

https://gist.github.com/spinningD20/f4e20c16b4cdf14e784d0eab67bd5a97

Should I try it on a fresh system that hasn't seen gamehub yet?

I also tried running gamehub after exporting the env vars above as you had specified, still a seg fault.

[hagabaka]

Thanks for helping with testing.

I also get a segfault if butler isn't installed. hagabaka/GameHub@af359d9 fixes this. In the mean time you can also try if installing butler (available on AUR) works.

[spinningD20]

https://gist.github.com/spinningD20/db4a7da0472a68c08c21c6a27bc7e96e

I am beyond the segfault now, thanks for the pointer about butler.

Observations so far:

• clicking on the itch.io "authentication is required" in the list of services when gamehub initially opens, nothing happens.
• I navigated to the options, found the itch.io section, clicked "Generate Key" and went through that process.
• Copy pasted the api key into the itch.io settings. status of service changed to Authenticated, and I saw my itch.io titles show up in the list of games.
• Right clicked "Baba Is You" (great game) and pressed Install. Install popup appeared, picked linux version, pushed Install.
• Baba Is You tile text displays "Starting Download", and noticed [FATAL] [Json] json_object_get_int_member: assertion 'node != NULL' failed in the terminal log. Nothing happens beyond this point.

[spinningD20]

I noticed a few different flags for running gamehub, and got it to give me some DEBUG level logging in the terminal. after the FATAL json null error, I got this json response:

https://gist.github.com/spinningD20/0411ec16bd1751f15dcf38d9b041a63f

The only thing I noticed that was null is the build value? Or perhaps this is some other (unrelated) response, I am not sure. Figured I would mention it!

EDIT: Also, I am running this directly from your fork's itch branch, hagabaka.

[hagabaka]

I reverted the commit which added game updates but unfortunately also causes this problem with some games. Can you try if the latest commit works?

Also, it might help with testing to know that after you've installed the itch branch one time, the setting schema will be updated, and you don't need to install it every time after rebuilding, but can just run the newly built binary directly.

[spinningD20]

we're getting further! :)

With the latest commit, I am able to download and install Baba Is You. I went and installed Celeste as well.

So with the happy path out of the way, I started going down some other scenarios... one of them is canceling the download after it is started. When I do this, it does technically stop the Download, but the UI state of the tile text is not updated, still shows me the (last update of) the download progress. If I close gamehub and reopen it, it correctly shows not installed, and will let me go install it.

The only other side-effect of starting the download, canceling, closing gamehub, and reopening and reinstalling, is that it won't clean up the canceled download:

Not sure what the priority is, if we want any of the above fixed before it is merged in or not? Testing this same scenario with GOG games, it uses a file in the GOG collection directory to store the downloaded data, and when canceled and re-started, it uses the same file (canceling the download does not make gamehub delete the progress it made in a previous download attempt).

More just letting you guys know this detail I noticed. Otherwise, it seems to work as intended!

(disclaimer, I've only installed/uninstalled/downloaded/cancelled download, I am sure there are other things to test too :)

[hagabaka]

With the latest commit on my branch, status should be updated correctly when cancelling an installation.

For clearing or reusing files from cancelled installations, I think there should be an easier way to let butler handle it, but I'm still trying to get the information from the Itch developer. I feel this could be done in a separate issue after the main features have been merged.

[spinningD20]

Thanks @hagabaka ! While I only have a few games to test with from itch.io (most purchases I've done through them are for art assets), I would say it's ready to merge, and we'll tackle more undiscovered, involved issues as they arise.

@tkashkin or hagabaka, just let me know if you'd like me to do anything further, if there's anything more I can do to help out.

[tkashkin]

I have merged it into dev branch.
It may need more tests covering various cases (with/without butler installed, authenticated/not authenticated, free/paid games, etc.).

[Weirdo1312]

I have merged it into dev branch.
It may need more tests covering various cases (with/without butler installed, authenticated/not authenticated, free/paid games, etc.).

clicking on itch io does not open up the login page

[WARN] [ButlerClient: err 2] {"id":2,"error":{"code":-32603,"message":"apiKey: cannot be blank.","data":{"butlerVersion":"v15.17.0, built on Aug 13 2019 @ 23:04:11, ref 94c194ef1888b0363e399983355bf981dee38ff8","stack":"apiKey: cannot be blank."}},"jsonrpc":"2.0"}

[tkashkin]

@BarFin There is no login page, you need to open Settings and set your itch.io API key.

[Weirdo1312]

@BarFin There is no login page, you need to open Settings and set your itch.io API key.

oh i see now it worked

[hagabaka]

Is it possible to make the login page open the itch.io settings page, or show an information dialog? I think we can do that in Itch.authenticate() when the API key is empty or null, but I'm not sure how to open the settings page or show a dialog.

[5310]

Thank you for Itch support! Linux natives games seem to run splendidly, at least all the ones I've tested so far, free or paid.

But Windows games don't seem to launch at all. I don't even get an "executable cannot be found message", and even after setting it manually I get no "running with compatibility layer" text like the other Proton games. Does the Itch service not have any support for this yet?

[tkashkin]

@5310 Yes, compatibility layers are not supported for itch games yet. GameHub does not run games directly and butler does not expose available game executables.

[onnyyonn]

As @spinningD20 has already pointed out in his comment, clicking on "authentication is required" in the list of services does nothing. The user has to generate an api key, the link for which is buried in the settings. This is confusing to an user who doesn't already know what to do. Can the behaviour of "authentication is required" tab be changed to that of the "generate key" link? Or opening a window explaining the steps may also be useful.

[captn3m0]

Copy pasted the api key into the itch.io settings. status of service changed to Authenticated, and I saw my itch.io titles show up in the list of games.

This doesn't seem to work for me. I've pasted the generated key, I get a notice "Some settings will be applied after application restart".

I restart GameHub, but the itch.io status still says Not Installed.

I've tried twice with different keys

[onnyyonn]

@captn3m0 I suspect (but not completely sure) that the Itch client (or Butler) has to be installed for Gamehub to move from Not Installed to Authentication Required and then it can access Itch via the key.

[captn3m0]

That was it! Seems to have fixed it for now, thanks.

It ought to be able to just run off my generated key though?

[RPGHank]

Since the itch client is open source is it possible to have itch support built in to Gamehub without having to install the Itch client?
Or is there something which makees this not possible?

[ghost]

It still says "not installed" after pasting the key and installing both itch and butler. Butler is soft-linked to /usr/bin and /usr/bin/local and itch was installed normally.
EDIT: This problem only happens on flatpak version. It works fine when installed from .deb file.

[librarianmage]

May I ask what the status of this is?

[Weirdo1312]

May I ask what the status of this is?

Implemented and i believe it should be closed;
people who are experiencing issues when using itch io in gamehub should open new issues