[Feature] Public URL/page for modders to trigger re-indexing

[HebaruSan]

Motivation

Some modders maintain backport releases, so users of older game versions can get the latest features. To have these indexed properly in CKAN, they have to:

1. Release one version
2. Wait for the bot to index it
3. Release the next version

Step 2 requires the modder to check manually whether the first version is indexed yet. If it isn't, all they can do is check again later. Since a bot cycle can take 3-4 hours 10-20 minutes in good times and multiple days hours in bad times, this is not a good work flow.

Suggestion

We have web hooks today that can trigger immediate re-indexing of a given module (mainly for SpaceDock). It would be nice if this could be made publicly accessible, so the steps could instead be:

1. Release one version
2. Tell the bot to index it
3. Release the next version
4. Tell the bot to index it (optional)

These steps could be completed in a matter of minutes instead of unknown hours.

Front end

I imagine something like a dropdown to choose a module and a button next to it labeled "Re-index". It could be its own standalone page, it could be added to the status page, or something else.

Back end

Probably makes sense to support URLs like

• https://status.ksp-ckan.space/reindex?mod=Astrogator

Caveats

We may need some kind of throttling or security on the back end to prevent this from being used to DoS the bot, either intentionally or otherwise. For example, once it became widely known, well-meaning but clueless fans of popular mods might misuse it in the misconception that it can make their favorite mods update quicker.

[techman83]

Beyond GitHub/Spacedock are there a large amount of Mods hosted elsewhere? I had envisioned a workflow that involved an OAuth flow to allow our bot to add a webhook to their repository. But I've never looked too hard at how complex that would be.

[HebaruSan]

Wow, Google found this with "ckan host pie chart" ! I can never remember which issue I posted it on:

So yes, this feature idea is focused mainly on GitHub. Maybe @DasSkelett's KSP-AVC schema validation project will turn up a good way to provide this.

[DasSkelett]

What a coincidence. A few days days ago I tumbled over the issue wher you posted this, and decided to update it a bit, for fun, but didn't post it because it wasn't really relevant. SD and GitHub have an even bigger share now:

I don't see something opening up a way for how to safely trigger webhooks with the schema validation yet, but we might get some ideas.

[techman83]

I think with the status page, we could have a "login via github" similar to Jenkins. Then it could match up your mods, and you could add a "Enable webhooks". We'd need to store the shared secret somewhere and to ensure that if it leaks we're not hosed, we can have a per mod secret randomly generated and stored in dynamodb, combined with a server secret stored separately. We'd need to fix the SSL on status.ksp-ckan.space, but that's doable.