Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move sync archive update into normal task schedule #211

Merged
merged 1 commit into from
Jan 4, 2021
Merged

Conversation

taldcroft
Copy link
Member

Description

This makes the sync archive update step run sequentially after the main archive update. This is better than separate jobs since it is robust when the main archive update takes longer than expected.

From some related investigation, it appears the time from comm EOT to L0 decom products appearing in the archive is less than 2 hours. EOT is very rarely past 2am, so we can do:

  • Eng archive task_schedule starting at 4am. Typical time to finish should be less than 2 hours.
  • Cheta_sync on cheru starting at 7:30am.

I looked into the cheta_sync code and it updates the archfiles.db3 (the master index file for a content type) last, so that I don't see an easy way for user fetch queries to fail even if done right in the middle of an update. Of course that doesn't mean it won't happen, so we'll revisit the cheru update timing if needed.

Testing

  • [N/A] Passes unit tests
  • [N/A] Functional testing

I don't see how to test on the side without just installing it. There is not much opportunity for this to fail except for file permissions.

We'll need to do a chmod -R g+w /proj/sot/ska/data/eng_archive/sync before promoting this.

@taldcroft taldcroft merged commit 04017f0 into master Jan 4, 2021
@taldcroft taldcroft deleted the task-sync branch January 4, 2021 16:32
@javierggt javierggt mentioned this pull request Mar 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant