As a node operator, I want the the registry schema to update autonomously when new data is ingested.

[jordanpadams]

For more information on how to populate this new feature request, see the PDS Wiki on User Story Development:

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Issue-Tracking#user-story-development

Motivation

...so that I can add new data to the registry without having to load new metadata schemas.

Additional Details

This will require some "central registry" containing all schemas that a local harvest or registry manager can query for information.

Acceptance Criteria

Given data containing attributes never before registered in the registry AND the attributes exist in a currently released LDD
When I perform a registry ingestion
Then I expect the registry to "figure out" how to update the schema appropriately, and ingest the data without error

Given data containing attributes never before registered in the registry AND the attributes do NOT exist in a currently released LDD
When I perform a registry ingestion
Then I expect the registry will raise a descriptive error noting how to update the schema. see #110

Tasks

• Create a CSV file listing all LDDs, for example,

"pds","https://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_JSON_1F00.JSON","1.15.0.0","2020-12-23T15:16:28Z"
"cart","https://pds.nasa.gov/pds4/cart/v1/PDS4_CART_1F00_1950.JSON","1.15.0.0","2020-12-21T21:48:19Z"
"insight","https://pds.nasa.gov/pds4/mission/insight/v1/PDS4_INSIGHT_1B00_1870.JSON","1.11.0.0","2019-08-02T21:49:46Z"
"orex","https://pds.nasa.gov/pds4/mission/orex/v1/orex_ldd_OREX_1400.JSON","1.7.0.0","2020-11-06T17:27:52Z"
...

• Deploy the CSV file "https://raw.githubusercontent.com/NASA-PDS/pds4-information-model/master/docs/ldds/pds4-ldd-config.csv"
• Registry Manager: Read LDD list CSV file from default and custom URL.
• Registry Manager: Change "load-dd" command to store current LDD version and date in Elasticsearch.
• Registry Manager: Change "load-data" command to automatically load LDDs

[jordanpadams]

@tdddblog for our Monday meeting, let's think about this ticket a bit. i think this is something we will need to improve the node user experience ingesting data into the registry. if it seems like it will take too much work, then maybe we will push it out, but I would like to think about how much effort this will take and if it is worth it.

[tloubrieu-jpl]

Questions for the breakout:

• where to store the map namespace - LDD location
• how to manage versions of LDD (minor version missing from the json file, would need to parse the filename).

[jordanpadams]

@tdddblog
PDS4_IMG_1G00_1830.JSON.txt

how does that look for the JSON?

[tdddblog]

@jordanpadams
That looks good. Will all LDDs including mission LDDs be updated?

[jordanpadams]

unfortunately no. only the discipline LDDs we will build for this release. so it is probably worth it to key off the date, even if we include that version number in the CSV we create.

[tloubrieu-jpl]

CSV is ready, needs to be uploaded to github. We need to discuss that during the breakout.

@tdddblog is also doing refactoring on the code writing/reading the LDD to registry so that it is re-usable in different part of the application.

@tdddblog is also writing the documentation in the code.

[tloubrieu-jpl]

The code is written, @tdddblog is writting comments before creating the pull request