feat: continue export in a new job after minutes-limit is reached

[shadinaif]

feat: continue export in a new job after minutes-limit is reached

The problem is that the new behavior of celery after the upgrade is to restart tasks automatically after 60 minutes. Many export tasks require more than an hour to finish. Therefore, we implement a continuing mechanism that creates a continue-job after a configured number of minutes and then terminates the current one.

Here is the flow:

• Export-Task created by the export API
• The Export-Task creates a celery-task to process the export
• Celery starts the celery-task
• The celery-task continually checks for elapsed time since the start of itself
• If minutes-limit is reached (which is 5 minutes by default):
  • The celery-task uploads the collected data to S3 in a partial file
  • The celery-task initiates a new celery-task with a mark that identifies the continuing point
  • The celery-task stops
  • Celery starts the new celery-task
• If the data is fully collected:
  • If this is a continuation celery-task:
    • The celery-task uploads the collected data to S3 in a partial file
    • The celery-task downloads all partial files into a single local file
    • The celery-task uploads the local file to S3 as the final exported file
    • The celery-task deletes all partial files from S3
  • If this is the first celery-task:
    • The celery-task uploads the collected data to S3 as the final exported file
  • The celery-task marks the Export-Task as completed