Allow re-use of StructuredFile data endpoints with external filters #944
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
File and RDBMS DataEndpoints use various pieces of information such as (type, path, mode) or (type, config, schema) to generate a unique key that can be used so we don't instantiate multiple copies of the same resource handle. However, StructuredFile endpoints allow data to be optionally filtered through one or more external applications such as jq and awk and these filters were not taken into account when generating the unique key. At some point in the past this was addressed by assigning a key to StructuredFile endpoints using a constant string and an increasing number. Adding new StructuredFile endpoints caused tests that checked this key to fail depending on the order in which the files were processed.
This has been addressed by taking the external filters into account when generating a unique key so a StructuredFile
person.json
and one that calls a filter such asjq 'map({name: .organizations[].name}) | unique' person.json
are treated as two distinct resources.Ignore commit "Comment reformating and cleanup" as it updates and re-formats comments to be more in line with current XDMoD documentation conventions.
Motivation and Context
Reduce the overhead when adding new tests and parsing multiple JSON files.
Tests performed
Yes.
Types of changes
Checklist: