Match metadata paths in JSON parser #55
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Match metadata paths in JSON parser
Problem
Node may run out of available memory if input JSON contains large top-level elements that are matched by the parser's
$.*
path.Solution
Specify paths for defined metadata elements instead of matching all top-level elements. This allows the parser to skip undefined elements, which reduces memory usage for parsing.
Result
Validator will be able to run on JSON files containing large top-level elements without exhausting available memory.
Test Plan
Code was tested using JSON HPT MRF that contained its standard charge information in a top-level element with a key not defined by the schema. The file was large enough such that prior to the change, Node ran out of heap space. Specifically, the file was 1GB, and Node was configured with the default heap size. After the change, the validator was able to finish running and return a result object.