Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow re-use of StructuredFile data endpoints with external filters #944

Merged
merged 6 commits into from
May 24, 2019

Conversation

smgallo
Copy link
Contributor

@smgallo smgallo commented May 23, 2019

Description

File and RDBMS DataEndpoints use various pieces of information such as (type, path, mode) or (type, config, schema) to generate a unique key that can be used so we don't instantiate multiple copies of the same resource handle. However, StructuredFile endpoints allow data to be optionally filtered through one or more external applications such as jq and awk and these filters were not taken into account when generating the unique key. At some point in the past this was addressed by assigning a key to StructuredFile endpoints using a constant string and an increasing number. Adding new StructuredFile endpoints caused tests that checked this key to fail depending on the order in which the files were processed.

This has been addressed by taking the external filters into account when generating a unique key so a StructuredFile person.json and one that calls a filter such as jq 'map({name: .organizations[].name}) | unique' person.json are treated as two distinct resources.

Ignore commit "Comment reformating and cleanup" as it updates and re-formats comments to be more in line with current XDMoD documentation conventions.

Motivation and Context

Reduce the overhead when adding new tests and parsing multiple JSON files.

Tests performed

Yes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@smgallo smgallo added enhancement Enhancement of the functionality of an existing feature Category:ETL Extract Transform Load labels May 23, 2019
@smgallo smgallo added this to the 8.5.0 milestone May 23, 2019
@smgallo smgallo force-pushed the etl/improve-structured-file-key branch from 431b6ac to eae677b Compare May 23, 2019 19:05
@smgallo smgallo requested review from ryanrath and plessbd May 23, 2019 19:40
@smgallo smgallo merged commit 88a22ad into ubccr:xdmod8.5 May 24, 2019
@smgallo smgallo deleted the etl/improve-structured-file-key branch May 24, 2019 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:ETL Extract Transform Load enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants