[Enhancement]: Regularly monitor peak and resident memory usage of the provider

[breathingdust]

Description

Recent issue #31722 has demonstrated that the increasing size of the provider, combined with the recent addition of resources with a deep and complex schema has significantly increased the peak memory requirements of using the provider. This has put pressure on memory requirements in practitioner infrastructure deploy pipelines and has resulted in OOM errors when making changes to their infrastructure.

While we work on optimizing memory requirements, we should also begin to actively monitor the memory footprint of the provider as it evolves. This is likely to consist of a scheduled GitHub action to obtain a memory profile of the running provider via an acceptance test, and visualizing the results via pprof.

We should also define thresholds for what is considered an acceptable, vs an anomalous increase. This should allow us to more proactively respond to provider changes which result in outsized memory requirements.

Affected Resource(s) and/or Data Source(s)

All

References

• https://github.com/google/pprof
• [Bug]: Memory consumption has increased since v4.67.0 #31722
• Significant memory usage at init with 4.29.0 provider #26742
• See memory use over time: https://github.com/YakDriver/awsprovmemprof
• Regex cache: https://github.com/YakDriver/regexache
• [Enhancement]: Improve memory allocation in the provider #33553

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[gavinelder]

@breathingdust can #26742 please be added as a reference , this is the provider in which we have observed the largest increase in memory consumption

[YakDriver]

We are regularly monitoring memory now and have accomplished reducing memory to an extent. For future travelers, here is a sum up.

What we've done

For this issue, we have:

• Implemented regular expression caching (de-duplicating), which reduces memory allocation by ~15-20%.
  • Use regex cache to save memory #33147
  • semgrep: Add regexache check #33156
  • lightsail: Fix regex name validation #33405
  • Regular expression consistency #33359
  • Use new version of regexache #33317
  • skaff, docs: Update for regexache #33164
  • https://github.com/YakDriver/regexache
• Created a daily job in TeamCity that reports changes in memory allocation, use, and CPU time from current to the most recent build. Below is an example of the output. Note: The bottom-line numbers do not themselves relate perfectly to running Terraform in the real world because they involve running in the test environment and running specific example tests, the average of one VPC and one SSM. However, they are useful in comparing iterations of the AWS provider to each other.
  • teamcity: Performance job #33434
  • teamcity/performance: New performance job #33530
  • teamcity/performance: Fix script #33538
  • https://github.com/YakDriver/awsprovmemprof

Performance changes from latest version (v5.17.0) to main
Allocated memory: 1.7% (5744.1MB to 5841.2MB) (Worse)  
In-use memory: 9.4% (24.0MB to 26.2MB) (Worse) (wide-fluctuations normal)
CPU time: 4.8% (29.0s to 30.4s) (Worse)

What remains to be done

See #33553.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.