Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: context handling performance & memory improvements #1336

Merged
merged 9 commits into from
Nov 5, 2024

Conversation

jensneuse
Copy link
Member

@jensneuse jensneuse commented Nov 5, 2024

Previously, we were parsing variables and passing them as byte slice to the engine. Then engine then parsed the variables into an astjson object and started fetching data. As we're loading data layer by layer, we had to create a "context" object of available data at this layer which can be used to render the input templates for each fetch.

With this rewrite, variables are immediately parsed into an astjson object. As we get new data from a fetch, we parse it into astjson format and merge it with the existing data at this layer. Then, we're not marshalling it into a json byte slice before passing it to the templating engine, but we're passing the astjson object directly, as the templating engine now accepts astjson format.

As a result, we're saving one marshalling of layer data per fetch, which reduces memory usage and saves some cpu cycles, as the templating engine also doesn't have to parse the json but can immediately access the ast json object.

@jensneuse jensneuse changed the title chore: cleanup ctx handling feat: context handling performance & memory improvements Nov 5, 2024
Copy link
Contributor

@StarpTech StarpTech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@jensneuse jensneuse merged commit 1aed39a into main Nov 5, 2024
11 checks passed
@jensneuse jensneuse deleted the feat/improve-ctx-variables-handling branch November 5, 2024 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants