Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate new provenance.py script into operational deployment to tag products with Provenance information #141

Closed
jordanpadams opened this issue Dec 16, 2022 · 3 comments

Comments

@jordanpadams
Copy link
Member

💡 Description

Follow-on to:

Need to setup cron or lambda function or ??? to run on semi-regular cadence (every 45 minutes?) to update the operational indexes with the appropriate superseded_by metadata.

Rationale for 45 minute cadence:
Per discussion on NASA-PDS/registry-api#441, the script takes ~15 minutes to run on a few million products, so 45 minutes should give us plenty of leeway. Additionally, we can then tell node folks to expect their products to be life through the API ~60 minutes after they tag the products for release using Registry Manager.

@tloubrieu-jpl
Copy link
Member

tloubrieu-jpl commented Jan 26, 2023

Steps to migrate:

  1. export schema for each of the discipline nodes
  2. recompute the new field values (here run provenance.py)
  3. create a temporary index and add/remove node specificities
  4. reindex the updated documents on the temporary index
  5. drop the registry index
  6. rename index

We need to re-index because the documents have been previously indexed (that sounds obvious, isn't it). What that does mean is: if the schema is updated that does not apply on document previously indexed.

This should be done in a script because that will need to be applied every build as we are expecting upgrades of the registry schema.

Overall we want to re-analyze how the provenance/all/latest feature is managed in the registry so to have an acceptable robustness/availability of the registries.

@jordanpadams
Copy link
Member Author

@jimmie wrapping up documentation today. then heading onto schema updates

@jordanpadams
Copy link
Member Author

all sub-tasks completed. demoed at 2/23 sprint review and a couple minor tweaks to be wrapped up today

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants