Performance Improvement Plan

[jpogran]

Extension Version

v2.27.0

Problem Statement

We've been investigating performance for terraform-ls to develop a short term and a long term plan to address the issues that have been reported.

We know most of the issues reported surround the work we do in the background to provide features like intellisense, go to references, hover documentation, etc. When you open up the editor, we do a lot of work in the background to understand the code you're working on. We have an indexing process that finds all the Terraform files and modules in your current working directory, parses them, and builds an understanding of all the interdependencies and references inside. We hold this information inside an in-memory database, which we update as you change your files. If you open a directory with many hundreds of folders and files, this may consume more CPU and memory than we intend.

Proposal

In the short term we're working to add ways to measure and record the performance of terraform-ls in different situations so we can pinpoint the best path forward. We need a way to record the time it takes to perform various tasks in a manner that will give us the information we need while also respecting user privacy and not intruding on usage. We’ve started to instrument opt-in only performance tracing that is currently only used by the developers on the team.

We're also reaching out to users who have reported issues to get a closer look at how they are using VS Code to edit Terraform code. We've noticed that people edit their Terraform code using many different approaches. Some open single files, some open single folders, others open a directory containing many hundreds of modules and Terraform files. No individual approach is wrong or right, but each requires some consideration when developing a language server.

We've implemented several targeted improvements over the last several point releases based on our investigations: (1332 and 1258 ). We’ve also added a language status indicator in #1547 that approximates an indicator when the language server is ready for use so you can see that there are things happening in the background.

Most of the work we anticipate doing is optimizing these background processes. We plan to use the performance data we collect to inform places where we can either optimize or do less work entirely #1339. We are also planning to add a configuration setting #1341 so that users can adjust/tune the amount of workers handling these background jobs to suit their individual needs.

This is by no means all we intend to do, but it is a start. We are hoping these set of steps will inform us on the best approach for resolving these issues long term.

This issue will be pinned to the repo and updated with links to work being done to address this as they are planned and taken on.

Shipped

• Stable extension 2.32.0

  • LS state & performance refactoring terraform-ls#1667

• Stable extension 2.27.2

  • Improve PreloadEmbeddedSchema performance terraform-ls#1369
  • langserver: Avoid re-processing open files after file saving terraform-ls#1372

Current Investigations

• Avoid re-processing files on textDocument/didOpen if content is the same as on disk terraform-ls#1375

References

• Open performance issues

Help Wanted

• I'm interested in contributing a fix myself

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[adrianisk]

Thanks for the update!

[jorhett]

I think some sensible recognition of root modules in large monorepos would greatly help. You don't need to rescan every file (traces show you are) in a huge monorepo while I'm working on a single module.

[FernandoMiguel]

I used to have one window with many workspaces and then 1 or 2 windows with a much smaller number of workspaces (usually a couple root modules and the modules they depend on).
Sadly I can't disable terraform on just one window as it seems to be a global plugin.

[jpogran]

We've just released 2.27.2 which contains two fixes (hashicorp/terraform-ls#1369, hashicorp/terraform-ls#1372) in our continuing efforts to address this issue.

With the fix, we should be back to 0.29.2 levels of CPU usage while maintaining the memory improvements of 0.29.3.

If you have the time, please give it a try and let us know how it works for you. Please open a new issue rather than replying here, as we're using this issue as a tracking issue.

[seanamos]

I just tested the changes in one of our older, large and monolithic terraform repos. The CPU usage and performance in the root of this workspace was terrible.
I'm seeing a dramatic improvement in both CPU usage and perceived responsiveness.

[jacoor]

I just installed v2.28.2023091812
There is huge improvements in file save time. CPU usage still too high, about same level, but at least I don't want 5 mins or more to see file saved. Now it takes ~ 15 seconds, even with error in the file.

[vinayhegde1990]

I found #1328 (comment) which led me to this central issue for tracking all performance debugging. My current setup has v2.28.2 of the hashicorp.terraform plugin for VSCode 1.83.1 which consumes ~1.4 GB RAM out of 8 GB, below is an image from the OS activity monitor

A workaround is to enable this plugin only during terraform development and disable it right afterwards. Seems this is not fixed yet, please let me know if any more information is required and if I could help fix this permanently.

[ghost]

As of v2.29.2, I'm still seeing ~10 GB of RAM usage by terraform-ls. Save performance has generally improved (through occasionally hangs).

[thatguythat1031]

I'm seeing CPU usage of 200%+ and memory usage of over 9GB (I've seen up to 13Gb) on 2.29.3. I've tried disabling Enhanced Validation and that seems to help, but only a little.

[baalimago]

I found this thread while looking for "terraform high CPU usage" and just wanted to pitch in some help:

I had the same problem and got notification from neovim (I know this is for vscode-terraform, but both use terraform-ls, which is probably what's causing performance issues) and I noticed that this was being spammed about 40-50 times per second:

[ERROR][2024-01-30 10:13:35] .../vim/lsp/rpc.lua:734    "rpc"   "terraform-ls"  "stderr"        '2024/01/30 10:13:35 jobs.go:140: JOBS: Enqueueing new job "517": "OpTypePreloadEmbeddedSchema" for {"file:///<project-path>/.terraform/modules/<module-name>.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder.builder"} (IsDirOpen: false, IgnoreState: false)\n'

And when I looked in the local .terraform/modules direectory, I found that somehow I had a gigantic amount of local modules, all following similar builder.builder.... pattern. Do note, I do have a "builder" module in my project, that I consistently target locally module "builder" { source "../../<module-dir>" ..., but something was very off. Not sure how it happened. Perhaps it's related to neovim somehow, not sure.

Anyways, TL;DR, if you're experiencing performance issues, have a peek in your lsp logfile and .terraform/modules file to see if anything strange is going on there. I cleared out the local .terraform/modules, and now the performance is spot on.

[ghost]

I'll add that enabling TF_PLUGIN_CACHE_DIR="~/.terraform.d/plugin-cache" also seems to help performance (though that could be coincidental).

[thatguythat1031]

My issue appears to be caused by an internal terraform modules repo we maintain. Many of our other repos use the internal modules repo, which results in every terraform directory in several of our large repos having a copy of that modules repo in its .terraform directory.

Deleting as many of those .terraform directories as I can brings down the memory usage significantly- I was able to reduce usage from 10gb to 5.6gb by deleting around 8 directories. But that's only a temporary solution and requires constant upkeep.

[dogmatic69]

Coming here from hashicorp/terraform-ls#1272, I'd guess my issue is similar to the comment just above.

My issue appears to be caused by an internal terraform modules repo we maintain. Many of our other repos use the internal modules repo, which results in every terraform directory in several of our large repos having a copy of that modules repo in its .terraform directory.

running latest stable ubuntu, terraform-ls version seems to be 0.32.8

Machine has 16Gb ram and terraform-ls process results in system using swap and subsequently becoming completely unresponsive.

[JBG-brad]

Running into the same problem with internal modules we maintain. Working on one of those modules on an M3 Mac Pro, I'm seeing 700% CPU usage and 24GB of RAM. It's made using the VSCode plugin for terraform completely impossible.

Folder structure is pretty small, just a subfolder for the module and the parent directory.

This came out of nowhere, I've made changes to this module for months and everything has been fine. Today; it's completely unusable.

The weird thing I'm noting is that my system is making millions of calls to terraform cloud, but we do NOT use it. Watching the calls via wireshark, there seems to be some sort of infinite recursion. It appears to be trying to sync to terraform cloud, failing, then trying again at an ever increasing rate.

[dbanck]

Hi @JBG-brad, I would love to get to the bottom of this, but so far we haven't been able to reproduce most of the shared cases of extreme resource usage.

We try to get the schema for public modules from the public registry (https://registry.terraform.io/), but that should only happen once and the result is cached. It shouldn't happen for internal modules, and it definitely shouldn't happen in recursion. So I'm very curious to know what's going on. Maybe this could lead us to a new insight and improve performance for everyone.

Can you share the full language server log output with me? Or, time permitting, could you join a call and show me the bug in action?
You can also GPG-encrypt it if you're worried about sensitive data in it. You can also email it privately to daniel.banck<at>hashicorp.com if you prefer.

If anyone else wants to share their log privately, or has an easily reproducible performance problem, please don't hesitate to contact me.

[gullit-cw]

my log file is more than 25MB repeating the same content. then I separated only a snippet of the [full log](Full log: https://gist.github.com/gullitmiranda/78d9bc59869c17a45846e99f9f999930), if it helps:

Launching language server: /Users/home/.cursor/extensions/hashicorp.terraform-2.29.5-darwin-arm64/bin/terraform-ls serve
Client: Stopped --> Starting
2024/04/01 17:16:24 serve_command.go:108: Starting terraform-ls 0.32.8
2024/04/01 17:16:24 service.go:106: Preparing new session ...
2024/04/01 17:16:24 langserver.go:102: Starting server (pid 92988; concurrency: 4) ...
2024/04/01 17:16:24 opts.go:215: Received request batch of size 1 (qlen=0)
2024/04/01 17:16:24 opts.go:215: Dequeued request batch of length 1 (qlen=0)
2024/04/01 17:16:24 rpc_logger.go:32: Incoming request for "initialize" (ID 0): {"processId":92969,"clientInfo":{"name":"Cursor","version":"1.86.2"},"locale":"en","rootPath":"/Users/home/Code/terraform","rootUri":"file:///Users/home/Code/terraform","capabilities":{"workspace":{"applyEdit":true,"workspaceEdit":{"documentChanges":true,"resourceOperations":["create","rename","delete"],"failureHandling":"textOnlyTransactional","normalizesLineEndings":true,"changeAnnotationSupport":{"groupsOnLabel":true}},"configuration":true,"didChangeWatchedFiles":{"dynamicRegistration":true,"relativePatternSupport":true},"symbol":{"dynamicRegistration":true,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"tagSupport":{"valueSet":[1]},"resolveSupport":{"properties":["location.range"]}},"codeLens":{"refreshSupport":true},"executeCommand":{"dynamicRegistration":true},"didChangeConfiguration":{"dynamicRegistration":true},"workspaceFolders":true,"semanticTokens":{"refreshSupport":true},"fileOperations":{"dynamicRegistration":true,"didCreate":true,"didRename":true,"didDelete":true,"willCreate":true,"willRename":true,"willDelete":true},"inlineValue":{"refreshSupport":true},"inlayHint":{"refreshSupport":true},"diagnostics":{"refreshSupport":true}},"textDocument":{"publishDiagnostics":{"relatedInformation":true,"versionSupport":false,"tagSupport":{"valueSet":[1,2]},"codeDescriptionSupport":true,"dataSupport":true},"synchronization":{"dynamicRegistration":true,"willSave":true,"willSaveWaitUntil":true,"didSave":true},"completion":{"dynamicRegistration":true,"contextSupport":true,"completionItem":{"snippetSupport":true,"commitCharactersSupport":true,"documentationFormat":["markdown","plaintext"],"deprecatedSupport":true,"preselectSupport":true,"tagSupport":{"valueSet":[1]},"insertReplaceSupport":true,"resolveSupport":{"properties":["documentation","detail","additionalTextEdits"]},"insertTextModeSupport":{"valueSet":[1,2]},"labelDetailsSupport":true},"insertTextMode":2,"completionItemKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]},"completionList":{"itemDefaults":["commitCharacters","editRange","insertTextFormat","insertTextMode"]}},"hover":{"dynamicRegistration":true,"contentFormat":["markdown","plaintext"]},"signatureHelp":{"dynamicRegistration":true,"signatureInformation":{"documentationFormat":["markdown","plaintext"],"parameterInformation":{"labelOffsetSupport":true},"activeParameterSupport":true},"contextSupport":true},"definition":{"dynamicRegistration":true,"linkSupport":true},"references":{"dynamicRegistration":true},"documentHighlight":{"dynamicRegistration":true},"documentSymbol":{"dynamicRegistration":true,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"hierarchicalDocumentSymbolSupport":true,"tagSupport":{"valueSet":[1]},"labelSupport":true},"codeAction":{"dynamicRegistration":true,"isPreferredSupport":true,"disabledSupport":true,"dataSupport":true,"resolveSupport":{"properties":["edit"]},"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"honorsChangeAnnotations":false},"codeLens":{"dynamicRegistration":true},"formatting":{"dynamicRegistration":true},"rangeFormatting":{"dynamicRegistration":true},"onTypeFormatting":{"dynamicRegistration":true},"rename":{"dynamicRegistration":true,"prepareSupport":true,"prepareSupportDefaultBehavior":1,"honorsChangeAnnotations":true},"documentLink":{"dynamicRegistration":true,"tooltipSupport":true},"typeDefinition":{"dynamicRegistration":true,"linkSupport":true},"implementation":{"dynamicRegistration":true,"linkSupport":true},"colorProvider":{"dynamicRegistration":true},"foldingRange":{"dynamicRegistration":true,"rangeLimit":5000,"lineFoldingOnly":true,"foldingRangeKind":{"valueSet":["comment","imports","region"]},"foldingRange":{"collapsedText":false}},"declaration":{"dynamicRegistration":true,"linkSupport":true},"selectionRange":{"dynamicRegistration":true},"callHierarchy":{"dynamicRegistration":true},"semanticTokens":{"dynamicRegistration":true,"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator","decorator","hcl-attrName","hcl-blockType","hcl-blockLabel","hcl-bool","hcl-string","hcl-number","hcl-objectKey","hcl-mapKey","hcl-keyword","hcl-referenceStep","hcl-typeComplex","hcl-typePrimitive","hcl-functionName"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary","hcl-dependent","terraform-data","terraform-locals","terraform-module","terraform-output","terraform-provider","terraform-resource","terraform-provisioner","terraform-connection","terraform-variable","terraform-terraform","terraform-backend","terraform-name","terraform-type","terraform-requiredProviders"],"formats":["relative"],"requests":{"range":true,"full":{"delta":true}},"multilineTokenSupport":false,"overlappingTokenSupport":false,"serverCancelSupport":true,"augmentsSyntaxTokens":true},"linkedEditingRange":{"dynamicRegistration":true},"typeHierarchy":{"dynamicRegistration":true},"inlineValue":{"dynamicRegistration":true},"inlayHint":{"dynamicRegistration":true,"resolveSupport":{"properties":["tooltip","textEdits","label.tooltip","label.location","label.command"]}},"diagnostic":{"dynamicRegistration":true,"relatedDocumentSupport":false}},"window":{"showMessage":{"messageActionItem":{"additionalPropertiesSupport":true}},"showDocument":{"support":true},"workDoneProgress":true},"general":{"staleRequestSupport":{"cancel":true,"retryOnContentModified":["textDocument/semanticTokens/full","textDocument/semanticTokens/range","textDocument/semanticTokens/full/delta"]},"regularExpressions":{"engine":"ECMAScript","version":"ES2020"},"markdown":{"parser":"marked","version":"1.1.0"},"positionEncodings":["utf-16"]},"notebookDocument":{"synchronization":{"dynamicRegistration":true,"executionSummarySupport":true}},"experimental":{"refreshModuleProvidersCommandId":"client.refreshModuleProviders","refreshModuleCallsCommandId":"client.refreshModuleCalls","telemetryVersion":1,"refreshTerraformVersionCommandId":"client.refreshTerraformVersion"}},"initializationOptions":{"validation":{"enableEnhancedValidation":false},"experimentalFeatures":{"validateOnSave":false,"prefillRequiredFields":false},"ignoreSingleFileWarning":false,"terraform":{"path":"","timeout":"","logFilePath":""},"indexing":{"ignorePaths":[],"ignoreDirectoryNames":[]}},"trace":"off","workspaceFolders":[{"uri":"file:///Users/home/Code/terraform","name":"terraform"},{"uri":"file:///Users/home/Code/terraform-modules","name":"terraform-modules"}],"workDoneToken":"12136c6f-e906-4a08-8332-b4e166d76827"}
2024/04/01 17:16:24 initialize.go:185: enabling telemetry (version: 1)
2024/04/01 17:16:24 initialize.go:190: telemetry enabled (version: 1)
2024/04/01 17:16:24 scheduler.go:56: launching eval loop 0
2024/04/01 17:16:24 service.go:482: started low priority scheduler
2024/04/01 17:16:24 scheduler.go:56: launching eval loop 0
2024/04/01 17:16:24 service.go:487: started high priority scheduler
2024/04/01 17:16:24 opts.go:215: Posting server notification "telemetry/event" {"v":1,"name":"initialize","properties":{"experimentalCapabilities.referenceCountCodeLens":false,"experimentalCapabilities.refreshModuleCalls":true,"experimentalCapabilities.refreshModuleProviders":true,"experimentalCapabilities.refreshTerraformVersion":true,"lsVersion":"0.32.8","options.commandPrefix":false,"options.excludeModulePaths":false,"options.experimentalFeatures.prefillRequiredFields":false,"options.experimentalFeatures.validateOnSave":false,"options.ignoreSingleFileWarning":false,"options.indexing.ignoreDirectoryNames":false,"options.indexing.ignorePaths":false,"options.rootModulePaths":false,"options.terraform.logFilePath":false,"options.terraform.path":false,"options.terraform.timeout":"","options.validation.earlyValidation":false,"root_uri":"dir"}}
2024/04/01 17:16:24 walker_paths.go:255: walking next dir: {"file:///Users/home/Code/terraform"}
2024/04/01 17:16:24 walker.go:212: skipping ignored dir name: .git
2024/04/01 17:16:24 rpc_logger.go:53: Response to "initialize" (ID 0): {"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":[".","["],"resolveProvider":true},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"declarationProvider":true,"definitionProvider":true,"referencesProvider":true,"documentSymbolProvider":true,"codeActionProvider":{"codeActionKinds":["source.formatAll.terraform"]},"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"executeCommandProvider":{"commands":["terraform-ls.module.callers","terraform-ls.module.calls","terraform-ls.module.providers","terraform-ls.module.terraform","terraform-ls.rootmodules","terraform-ls.terraform.init","terraform-ls.terraform.validate"],"workDoneProgress":true},"semanticTokensProvider":{"legend":{"tokenTypes":["enumMember","function","keyword","number","parameter","property","string","type","variable","hcl-attrName","hcl-blockType","hcl-blockLabel","hcl-bool","hcl-string","hcl-number","hcl-objectKey","hcl-mapKey","hcl-keyword","hcl-referenceStep","hcl-typeComplex","hcl-typePrimitive","hcl-functionName"],"tokenModifiers":["defaultLibrary","hcl-dependent","terraform-backend","terraform-connection","terraform-data","terraform-locals","terraform-module","terraform-name","terraform-output","terraform-provider","terraform-provisioner","terraform-requiredProviders","terraform-resource","terraform-terraform","terraform-type","terraform-variable"]},"full":true},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"},"fileOperations":{}},"experimental":{"referenceCountCodeLens":false,"refreshModuleProviders":true,"refreshModuleCalls":true,"refreshTerraformVersion":true}},"serverInfo":{"name":"terraform-ls","version":"0.32.8"}}
2024/04/01 17:16:24 opts.go:215: Completed 1 requests [9.152ms elapsed]
2024/04/01 17:16:24 walker.go:242: walking of {file:///Users/home/Code/terraform/.github/workflows} finished
2024/04/01 17:16:24 walker.go:242: walking of {file:///Users/home/Code/terraform/.github} finished
2024/04/01 17:16:24 walker.go:212: skipping ignored dir name: .git
2024/04/01 17:16:24 walker.go:242: walking of {file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/.github} finished
2024/04/01 17:16:24 walker.go:218: found module {file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns}
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "1": "OpTypeParseModuleConfiguration" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "2": "OpTypeLoadModuleMetadata" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "3": "OpTypeParseVariables" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "4": "OpTypeDecodeVarsReferences" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 walker.go:91: parsed datadir: &datadir.DataDir{ModuleManifestPath:"", PluginLockFilePath:""}
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "5": "OpTypePreloadEmbeddedSchema" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "6": "OpTypeDecodeReferenceTargets" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "7": "OpTypeDecodeReferenceOrigins" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 jobs.go:321: JOBS: Dispatching next job "1" (scheduler prio: -1, job prio: 0, isDirOpen: false): "OpTypeParseModuleConfiguration" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns"}
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 walker.go:218: found module {file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test}
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "8": "OpTypeParseModuleConfiguration" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "9": "OpTypeLoadModuleMetadata" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "10": "OpTypeParseVariables" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "11": "OpTypeDecodeVarsReferences" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 walker.go:91: parsed datadir: &datadir.DataDir{ModuleManifestPath:"", PluginLockFilePath:""}
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "12": "OpTypePreloadEmbeddedSchema" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "13": "OpTypeDecodeReferenceTargets" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:140: JOBS: Enqueueing new job "14": "OpTypeDecodeReferenceOrigins" for {"file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test"} (IsDirOpen: false, IgnoreState: false)
2024/04/01 17:16:24 jobs.go:299: retrying on obj is nil
2024/04/01 17:16:24 walker.go:242: walking of {file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns/test} finished
2024/04/01 17:16:24 walker.go:242: walking of {file:///Users/home/Code/terraform/applications/active-directory/.terraform/modules/example_com/cloudflare/dns} finished

Full log: https://gist.github.com/gullitmiranda/78d9bc59869c17a45846e99f9f999930

[JBG-brad]

Hi @JBG-brad, I would love to get to the bottom of this, but so far we haven't been able to reproduce most of the shared cases of extreme resource usage.

We try to get the schema for public modules from the public registry (https://registry.terraform.io/), but that should only happen once and the result is cached. It shouldn't happen for internal modules, and it definitely shouldn't happen in recursion. So I'm very curious to know what's going on. Maybe this could lead us to a new insight and improve performance for everyone.

Can you share the full language server log output with me? Or, time permitting, could you join a call and show me the bug in action? You can also GPG-encrypt it if you're worried about sensitive data in it. You can also email it privately to daniel.banck<at>hashicorp.com if you prefer.

If anyone else wants to share their log privately, or has an easily reproducible performance problem, please don't hesitate to contact me.

I'd be happy to jump on a call and show you what's going on. I believe my email is in my profile.

[dbanck]

We've just released v2.31.2024061114, which ships with a re-architected language server for improved performance and resource usage (terraform-ls#1667). The language server will now start up much faster and use fewer resources, especially on larger workspaces.

In VS Code, you can opt in to the extension preview:

If you have the time, please give it a try and let us know how it works for you. We have tested this with workspaces of different sizes, but still expect some bugs due to the wide variety of configurations. Your feedback is greatly appreciated. Please open a new issue rather than replying here, as we're using this issue as a tracking issue.

[FernandoMiguel]

@dbanck i've open all the projects I have in my drive and seems to be behaving much better than ever before

[dogmatic69]

I've just installed it and restarted everything. Opened up some of my bigger projects and much better. Comfortably opened 4 projects 🫢

VS Code is just about snappy again, opening a new project takes a few seconds and does not completely choke up my machine 🚀

[adamrothman]

I can confirm that performance is massively improved using the pre-release version – thank you very much to all involved! Awesome work.

[vinayhegde1990]

I've also updated to the pre-release version v2.31.2024061114 and the memory usage is now between 30-40 MB on the OS activity monitor, a thorough 90-95% reduction from the previous version.

thank you very much to everyone involved, great stuff 👏🏻

[migueleliasweb]

Sadly, for bigger projects (100k lines), I feel the Terraform extension still tries way too hard and must be looking to parse every single file recursively - this is the only possibility I can think of.

I'm still seeing the language server taking over 1 minute to return even after adding a single character to an existing string.

I can sometimes see CPU spikes of 450%+ and a lot more memory than in this picture 😿 .

[dbanck]

@migueleliasweb Can you double check if you're using the preview version of the extension? Usually the path to terraform-ls should contain the exact version number, in this case .../extensions/hashicorp.terraform-2.31.2024061114-linux-x64/bin/.... The path in your screenshot looks like the stable version

[migueleliasweb]

Trying now.

First impressions (30s of it) is that the "time to first interaction" feels faster and it seems to be using less memory.

I'll post if I find anything. 🫡

I didn't know this thread was specifically abou the preview given the last stable was released 4 days ago. My bad.

Edit: "onSaves" are now taking a few seconds instead of enough time to cook a bowl of rice 🎉 . I like this.

[dogmatic69]

The original issue was about performance, an update in preview was released 2 weeks ago

[dbanck]

We've shipped the re-architected language server in the 2.32.0 stable release last week. The language server will (still) start up much faster and use fewer resources, especially on larger workspaces.

Please let us know if you experience any problems or regressions.

[kunickiaj]

I think some sensible recognition of root modules in large monorepos would greatly help. You don't need to rescan every file (traces show you are) in a huge monorepo while I'm working on a single module.

Yep.. ours has 37.5k terraform files and unless you open a subfolder only in VSCode performance will be unusable... can't even save a file.

[tyrken]

Our terraform monorepo has only 1400 files (some of those are symlinked'ed repeats) over 110 folders, but I've have to give up on the VSCode extension two weeks ago. It took so long to save & format files (20-30 seconds) that I occasionally planned/applied on stale file content instead of the intended updated file, i.e. the extension was causing operational mistakes/bugs.

I fear a lot of "nice to have" features have been added as the expense of performance of the core stuff - syntax highlighting & auto-formatting. When we can run terraform fmt -recursive that checks the whole tree from scratch in less than 4 seconds, how come VSCode is taking so long on warm pre-processed data?