Background task sequencing issue causing download/indexing loops

[Jreuningschererhubbell]

Describe the bug
Sequencing error in background tasks causes repeated deletion and re-downloading/indexing of remote repos provided in config.toml.

This seems to have arisen as a side effect of the fix for issue #3692 that was implemented in #3703.

Information about your version
Main branch, as of Jan 17.

Information about your GPU
Apple M2 Max

Additional context

The fix for issue #3692 that was implemented in #3703 seems to have created a sequencing error.

When the hourly task runs, repos are scheduled for indexing. When the garbage collector runs, it does not see the repos in config.toml, so it removes them from the index and the repositories folder. Then when the scheduled indexing occurs, the repos are not found and so they are pulled again. This behavior is shown in the attached log file. Note that I modified the background task to run the hourly task every 6 minutes to increase testing speed. However, I also observed this behavior with the unmodified interval.

IndexLoop1.log

I managed to resolve the issue by ensuring the function list_all_code_repository included the repos in config.toml. This change is implemented in this fork.

[wsxiaoys]

Thank you for bringing this issue to our attention. It appears that a step was inadvertently omitted during the last fix, as detailed in #3703. We are currently addressing this oversight with a new fix in #3721.