Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup: Validation of macros-only Manifest in parse_node is very time-consuming #1098

Closed
cmcarthur opened this issue Oct 29, 2018 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@cmcarthur
Copy link
Member

cmcarthur commented Oct 29, 2018

@jthandy noticed that on a project with hundreds of models and hundreds of tests, parsing alone can take a long time. I asked him to profile the project in question, and it turns out that around 30-35 seconds is spent in parsing nodes alone. A very large component of that parsing is building and validating the macros-only Manifest in BaseParser.parse_node (at time of writing parsers/base.py line 107).

Profile result indicates each Manifest.__init__ call takes about 45ms. The vast majority (>90%) of that is validation. In a project with 750 models + tests, that adds up to about 34 seconds added to a run that we could strip off.

I can imagine some potential fixes:

  • we could cache the macros-only Manifest that gets re-used in parse_node
  • we could change parse_node to use the jinja AST to pull out refs and config values
  • we could skip validation when building the "fake" macros-only Manifest

Any one of these would result in an enormous speedup for the project in question.

@cmcarthur cmcarthur added the enhancement New feature or request label Oct 29, 2018
@drewbanin drewbanin modified the milestones: Stephen Girard, Grace Kelly Nov 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants