Updating Toolforge web service in anticipation of infrastructure changes

[whym]

Description

Does anyone actively maintain the toolforge instance called "commons-android-app"? Toolforge's admins asked to migrate it to Kubernetes: https://phabricator.wikimedia.org/T319638 (Actually, we only have to deal with a simple wrapper script usually, instead of directly dealing with Kubernetes.) What is the current status regarding the migration?

@maskaravivek and @madhurgupta10 - your names are listed in the link above.

I believe this is related to #2430 and in use.

If needed, I think I can help because I recently did it for other Toolforge instances, archiving and commons-app-stats - both written mainly in Python.

There is no concrete end date for the current grid infrastructure as far as I can see, but apparently it's been more than a year since the admins first started asking us to migrate.

[nicolas-raoul]

Thanks a lot for the hint!
I think nobody is maintaining it currently unfortunately.
If you are willing to help perform the migration, that would be super helpful! :-)

[maskaravivek]

Hi @whym Yes, it would be great if you could help migrate it.

[whym]

@maskaravivek Thank you for the response. I'd like to see what I can do, then. Can you add my account as a maintainer of the tool? It's "Whym" on Toolforge, too.
I think you can do that at https://toolsadmin.wikimedia.org/tools/id/commons-android-app .

Also, where is the latest code hosted at?

[nicolas-raoul]

This is supposed to be the latest code:
https://github.com/commons-app/commonsmisc

I don't think I have admin rights.

[maskaravivek]

@whym Added

[whym]

Thanks a lot! I confirmed that I have access to it now on Toolforge.

From the inside, it points to https://gitlab.wikimedia.org/toolforge-repos/commons-android-app which looks like a fork of the repository mentioned above. I have requested access to it too (using GitLab's functionality for that).

[sivaraam]

@whym I suppose you're already keeping track of the upcoming deadline for migrating the toolforge application that is used by the commons-app. I've assigned the corresponding Phabricator task to you in order to keep that task up to date.

Thanks for you help on this! 🙂
Let us know if you're facing any blocker and need help with making progress on this. We'll try to help in ways we can.

[whym]

I have not been able to spend much time for the Toolforge issue, sorry. I really hope to get it done, or at least have a tentative fix, by the February deadline. I should have some time after the end of January.

I'm confused by the fact that there seems to be 2 tools (or backend APIs) in use by the app currently. Tool A is available at https://tools.wmflabs.org/urbanecmbot/commonsmisc. Tool B is at https://tools.wmflabs.org/commons-android-app/tool-commons-android-app. I believe the GitLab repository I mentioned earlier contains the source code of the Tool B.

Is Tool B a super set, or an updated version of Tool A? It look like that is the case, but I cannot verify because .py files of Tool A are not readable. If yes, we can simply stop using Tool A (but I'm not sure why nobody did that, if that is the case).

The latter is called "Test" in the app's source code, as in

apps-android-commons/app/src/main/java/fr/free/nrw/commons/di/NetworkingModule.java

Lines 48 to 50 in 38b1a01

	private static final String TOOLS_FORGE_URL = "https://tools.wmflabs.org/urbanecmbot/commonsmisc";
	private static final String TEST_TOOLS_FORGE_URL = "https://tools.wmflabs.org/commons-android-app/tool-commons-android-app";

[nicolas-raoul]

@urbanecm Are the .py files powering https://tools.wmflabs.org/urbanecmbot/commonsmisc visible somewhere? Thanks! :-)

[sivaraam]

I have not been able to spend much time for the Toolforge issue, sorry. I really hope to get it done, or at least have a tentative fix, by the February deadline. I should have some time after the end of January.

No worries. Thanks for your help! 🙂

Is Tool B a super set, or an updated version of Tool A? It look like that is the case, but I cannot verify because .py files of Tool A are not readable. If yes, we can simply stop using Tool A (but I'm not sure why nobody did that, if that is the case).

I think the source code for "Tool A" could be found here: https://github.com/commons-app/commonsmisc

[urbanecm]

@urbanecm Are the .py files powering https://tools.wmflabs.org/urbanecmbot/commonsmisc visible somewhere? Thanks! :-)

As it was linked here, it comes from commons-app/commonsmisc here at GitLab indeed. I pushed a local hack I had available at Toolforge, just in case. I don't have any information about https://commons-android-app.toolforge.org/tool-commons-android-app/.

It looks like you're migrating this to a tool you all have maintainer access to. That's of course fine – if you want me to redirect the urbanecmbot URI somewhere else at any point, please do let me know.

[sivaraam]

I suppose commons-android-app toolforge instance came into existence when working on the app's leaderboard feature. We have some documentation of the DB schema and the various APIs it provides in our documentation repo.

Why it was done using a different toolforge instance rather than using the existing one is something that I'm not aware of. May be @maskaravivek and @madhurgupta10 might be able to chime in and shed some light on that.

In any case, thanks a lot for your understanding @urbanecm 🙂

[madhurgupta10]

@sivaraam As far as I can recall, we used a different instance for the leaderboard because it was easier to deploy to tool-commons-android-app than commonsmisc

[whym]

Thanks a lot for chiming in, @urbanecm @madhurgupta10. I think it might be wise to migrate away from urbanecm's tool instance, in the long run, that is, for the access issue you mentioned. But for now, I'm happy to keep the way it is. Maybe that is something we can continue discussing in a separate task.

I believe the migration of the "commons-android-app" tool is mostly done. Please let me know if something stopped working around the leaderboard. I see that contributions tab is not working properly - this seems to have to do with the fact that the urbanecmbot tool is not up - I'm talking about it with Urbanecm. But if something is wrong with the leaderboard, that's probably caused by the migration I did.

[sivaraam]

I believe the migration of the "commons-android-app" tool is mostly done.

That's great to hear! Thank you for your help, @whym 🙂

Please let me know if something stopped working around the leaderboard. I see that contributions tab is not working properly - this seems to have to do with the fact that the urbanecmbot tool is not up - I'm talking about it with Urbanecm.

Indeed. That seems to be the case for a few days now. We've switched to using the commons-android-app instance to address this but that change hasn't yet been released. You could use the app build from the main branch to verify your changes. If at all you need an APK, you could use the one I built from main just today [link].

But if something is wrong with the leaderboard, that's probably caused by the migration I did.

I just tried out and Leaderboard seems to be working fine for me. There's the caveat that I'm not an ardent user of that screen, thought. @nicolas-raoul or others may be able to be able to give more precise feedback 🙂

In any case, thanks a lot again for helping out!

[whym]

If at all you need an APK, you could use the one I built from main just today

Thanks a lot. I tried it and it seems to work well, including the contributions tab. A positive sign that we can start taking steps to merging the two Commons app tools, perhaps sooner than I thought.

[nicolas-raoul]

I can confirm that the leaderboard is working great! :-)

[whym]

It looks like we can close this, thanks everyone.

There are some remaining tasks I found during the migration (but I think these can be worked on later individually):

• Have a clear license (ongoing at Create LICENSE.txt commonsmisc#23)
• Commit the changes: Commit the changes made at Toolforge in February 2024 into the repository commonsmisc#24
• Add protection against abuses (I have some more concrete ideas about those abuses but won't go into details here for obvious reasons)
• Refactor the scripts - we can serve endpoints from one script, rather than having one file per end point (This might need some more discussion)

[sivaraam]

Thank you very much for helping with this @whym ! 🙂