Turn on performance improvements by default

[jtcohen6]

Namely:

• partial parsing
• experimental parser

This will require accounting for all known "fast-and-incorrect" edge cases and either (or both) of:

• fixing them
• documenting them, and what to do when they happen

This does not require accounting for all known "slow-but-correct" edge cases. While strictly better, those can come later, and they won't yield incorrect results in the meantime.

[leahwicz]

• Can we expand the experimental parsing scope?
• Depends on the results we get back from metrics
• Would have to update the tests as well

[leahwicz]

Where is the flag for this? In the profile?

[leahwicz]

Let's split this into 2 different tickets:

• Experimental parsing -> would be a 1
• Partial parsing -> would be a 4-8 with test updates and some additional code changes

[jtcohen6]

Where is the flag for this? In the profile?

Yes! I think CLI flag + profile + env var (+ other configuration mechanism?). I'm looking at our discussion in #2990 re: coordinating configs.

Can we expand the experimental parsing scope?

One thing I had discussed with @nathaniel-may last week: rather than simply setting use_experimental_parser: True for everyone, I think we'll actually want to:

• Move proven parts of the experimental parser into the default parser (and delete old default code) — it's no longer experimental, everyone gets to have it
• Keep around the use_experimental_parser flag/config/var, still False by default, as a way for users to always opt into the next phase of not-yet-proven experimentation

Noting this here because it's different from the way we'd turn on partial parsing by default—that we'd more straightforwardly accomplish by setting the default value of partial_parse to True.

[leahwicz]

The experimental parser remaining work is in this ticket: #3377

[jtcohen6]

Remaining must-haves before we turn these on by default. We can fix/handle (ideal), or document limitations with path to turn off (less desirable).

• Partial parsing "fast-and-incorrect" edge cases:

  • generate_x_name macro changes (Trigger full reparse if changes to widely used built-in macros #3810)
  • env vars needed at parse time (ref/source/config or yaml property) have changed
  • volatile variables (run_started_at, invocation_id, etc) needed at parse time (Jinja functions remain static in YAML files on RPC until modified #2330)

• Experimental parser:

  • Detect if user has overridden ref/source/config macros; if so, disable experimental parser

As far as actually turning on:

• Partial parsing: set to true by default. Let's also move all partial parsing logging from info-level to debug-level.
• Experimental parser: bake current use into default model parser; keep around flag/config (false by default) to use with iterative improvements + sampling in the future

[leahwicz]

@gshank @nathaniel-may I broke this story down into 2:

• Turn on partial parser by default #3867
• Turn on experimental parser by default #3866