Enhancement: Do existence check for linked articles instead of downloads

[mandolyte]

Consider adding "existence checks" as an option.

Rationale: Would gain speed and retain benefit of checking linked articles, but not checking the content of the linked article.

Method:
This Gitea API returns a JSON structure that can be kept in memory to check if a TW article exists:

https://git.door43.org/api/v1/repos/unfoldingword/en_tw/git/trees/master?recursive=true&per_page=99999

[RobH123]

Will definitely investigate this -- thanks @mandolyte and @richmahn!

[RobH123]

Hmmh, even adding one more 9 to that, still can't fetch entire tree for en_ugl -- only fetches the first set out of the 29,969 total entries! Will have to work out how to loop to get the next page(s) and then how to combine the JSON!

[RobH123]

Ok, it seems that Gitea supplies a maximum of 12,000 entries at one time. I was a bit confused that the truncated flag is set even for the last page, so it really seems to mean not all entries are there in this fetch rather than more entries still to come after these ones.

[mandolyte]

That max is configurable... what number would work so that pagination could be avoided? You can discuss with Rich Cecil

…

On Wed, Oct 20, 2021 at 2:33 AM Robert ***@***.***> wrote: Ok, it seems that Gitea supplies a maximum of 12,000 entries at one time. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2ZXLZVA6I65GTI2O4LQLUHZPD3ANCNFSM5F5AQH7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[richmahn]

It is configurable, but for the whole API. It is usually best practice to avoid making the server serve everything but rather just keep querying and append results to an array until you get no results (or truncated is false, as mentioned above). It also tells you the total count when querying the first page, so you can also know when you have them all by that.

[richmahn]

Actually it looks like you will need to get to a page that does not return anything (i.e. "tree": null) to get "truncated": false. Not sure if that is common API procedure or not. I didn't write this.

[RobH123]

@richmahn cc @mandolyte Yes, I ended up writing an append loop and that wasn't hard once I realised that even the last page has truncated set to false (as mentioned above).