Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Match metadata paths in JSON parser #55

Merged
merged 1 commit into from
Aug 30, 2024
Merged

Conversation

mint-thompson
Copy link
Collaborator

Match metadata paths in JSON parser

Problem

Node may run out of available memory if input JSON contains large top-level elements that are matched by the parser's $.* path.

Solution

Specify paths for defined metadata elements instead of matching all top-level elements. This allows the parser to skip undefined elements, which reduces memory usage for parsing.

Result

Validator will be able to run on JSON files containing large top-level elements without exhausting available memory.

Test Plan

Code was tested using JSON HPT MRF that contained its standard charge information in a top-level element with a key not defined by the schema. The file was large enough such that prior to the change, Node ran out of heap space. Specifically, the file was 1GB, and Node was configured with the default heap size. After the change, the validator was able to finish running and return a result object.

Specify paths for defined metadata elements instead of matching all
top-level elements. This allows the parser to skip undefined elements,
which speeds up parsing.
Copy link
Collaborator

@shaselton-usds shaselton-usds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me.

@mint-thompson mint-thompson merged commit cfed514 into main Aug 30, 2024
4 checks passed
@mint-thompson mint-thompson deleted the match-metadata-paths branch August 30, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants