High CPU usage by terraform-ls serve on save+format

[leynebe]

Versions

Extension

v2.25.2

VS Code

➜  ~ vscodium -v
1.74.3 f8b9aeda40058455453f1298fd606caf6ff8d675 x64

But I upgraded from 1.71.x to 1.74.3 to try to fix the issue but it didn't help.

Operating System

Linux hostname 6.1.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 07 Jan 2023 15:10:07 +0000 x86_64 GNU/Linux
EndeavorOS

Terraform Version

➜  ~ chtf
   0.13.7
   0.14.11
➜  ~ terraform -v
Terraform v1.3.7
on linux_amd64

Steps To Reproduce

Steps to reproduce the behavior:

1. Open vscode.
2. Try to save a modified .tf file with bad formatting.
3. See vscode taking a long time saving + formatting the file.
   
4. Using htop I can see the extension bombarding my cpu with terraform-ls serve processes.
   

Expected Behavior

Saves and formats file in milliseconds.

Actual Behavior

Never formats or saves and noticably slows down pc a lot.

Additional context

This started happening throughout the day. It might have to do with specific code but I cannot share.
I tried:

• upgrading vscodium to 1.74.3
• disabling and enabling and uninstalling and installing the extension
• restarting vscodium
• rebooting

but none of those had an effect.

For now I've had to disable the extension entirely and use another one since it is so debilitating to the system's performance.

[mhulscher]

This started happening to me recently as well, and it's not just format-on-save. Whenever I start typing, it takes a second for code to syntax-check each newly written character. I can see it moving slowly because it adds a squiggly red line to each character, one-by-one, at a snail's pace.

[jwshieldsGetty]

Adding my own experience here.
I have a multi-root Workspace in VSCode, one of the directories that I have as part of the workspace contains a large number of repositories from our internal Git hosting system.

PS C:\src\Git> (gci -Recurse -Directory -Hidden -Filter '.git') | Measure-Object
Count             : 1728

I find that upon opening VSCode, the plugin seems to walk every directory recursively, and also kick off a job which spews hundreds of thousands of lines containing getting provider schema
It seems that while this is all happening, the format on save, or even, format document function, do not work. Often times when saving, VSC will report "getting code actions" and spin for minutes on end, before ending up doing nothing. VSC will do this for over an hour, I've never timed it fully, but it seems after some arbitrary time, it stops doing all this, and occasionally starts allowing me to save files, while it formats them too. Other times it seems to give up or just quit working altogether.

Another note, I have seen this behavior with both the "mainline" and "preview" versions of this extension.

Similar to OP, I have tried updating VSC multiple times, enabling and disabling the extension, trying the preview vs main versions, rebooting, adding folders to the exclusion list of the extension (which it seems needs an exact match and isn't globbing or wildcarding? That's near impossible for me to even try to attempt to add to my config to exclude dirs to walk.)

VS Code Info:

Version: 1.76.0-insider (system setup)
Commit: 25514d899f4bd877803defb213719f8af3d997d3
Date: 2023-02-07T05:22:19.428Z
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: 10.2.154.23-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: No

[jwshieldsGetty]

Sorry for a second comment directly following my first, could this be related to the problems noted in this issue -> hashicorp/terraform-ls#1100 ?

[smiller171]

Could be. Personally I found my issue was directly related to the Codelens: Reference Count setting being enabled, in a project with an EXTREME number of variables.

[jwshieldsGetty]

Could be. Personally I found my issue was directly related to the Codelens: Reference Count setting being enabled, in a project with an EXTREME number of variables.

FWIW, I just tested/am testing right now. Disabled all "Codelens Reference Count" settings in VSC (including non-Terraform related options)
My CPU was pegged at 100% and VSC was spinning while the TF extension walked my directories, along with more messages around "getting schema"
That said, it does seem that all of the activity died down faster than it usually does. (Around an hour with references enabled, maybe 15 minutes without? Non-scientific numbers, I didn't calculate exact timings.)

This is anecdotal and my only attempt at reloading/using/etc VSC since trying your suggestion @smiller171, but I thought I'd throw it in.

[czomo]

Facing the same issue with HashiCorp Terraform v2.25.4. My MacOS is unusable while terraform-ls run in background.

[antonio-robles-1]

Same issue as reported here. In my case just a workspace with several terraform repositories, just starting vscode doing nothing special and getting this situation in the image.

[dbanck]

Thank you for taking the time to report this issue.

We released the new extension 2.26.0 version (and 0.31.1 language server version) with a potential fix. So please try it and let us know if it solves this issue.

[antonio-robles-1]

Pretty similar situation here, sorry for the bad feedback 😞

[radeksimko]

We plan to make some changes as part of hashicorp/terraform-ls#1056 to help us understand situations like this (performance). While CPU/memory profiles can already be collected, these would not be as useful without context (such as some understanding of the workspace - no of files, folders, sizes of files etc.).

In the meantime, the only suggestion I can make is to avoid opening large/deep workspaces with lots of folders, esp. monorepos - i.e. open the one or few individual folders that you actually need to edit at any given time. This may not be most convenient, but should help avoid most performance issues for now.

Thank you for your patience and understanding.

[scott-doyland-burrows]

I find that if I run a terraform init which in turn populates the .terraform directory, that this is when terraform-ls uses high CPU for minutes at a time. I don't need terraform-ls trawling through .terraform and I have tried to exclude this directory from being accessed, but then the extension will not start.

I don't know if allowing .terraform to be excluded would help with the CPU issue?

Here are logs that occur when I attempt to exclude .terraform

Failure to start terraform-ls. Please check your configuration settings and reload this window
Client: Starting --> Stopped
[Error - 4:34:03 PM] terraform client: couldn't create connection to server.
  Message: cannot ignore directory ".terraform"
  Code: -32098 

[stakauskas]

Same here. I have a folder structure in a repository with hundreds of .terraform directories, and when init happens in those, vscode starts to be unusable, as well as system while terraform-ls gets high CPU usage.

If, the same repository has not done terraform init yet - it's perfectly fine.

[rmccarthy-ellevation]

Is there going to be a fix for the terraform-ls high cpu issue?

[jgspratt]

Just switched to VS Code today. Two different plugins caused such high CPU that my Mac began losing charge and eventually ran out of battery while plugged in. This was one of them. I haven't modified any terraform files since switching: I've just pointed VS at my existing code root and after I installed this plugin, I saw the terraform-ls go nuts.

[garett-eiq]

I had this issue as well. Turning off the reference count setting like @smiller171 mentioned lowered CPU usage significantly (from 600% to around 150%), but using 150% CPU while my Terraform workspaces are sitting idle still seems excessive.

[jgspratt]

Today, I did the following:

1. Reverted to the stable release (v2.27.0), restarted UI
2. Ensured my Terraform > Codelens: Reference Count setting was disabled
3. Re-enabled language server
4. Restarted UI
5. Installed Terraform v1.5.3 x64 via Zip download for my Mac (I was coming from v1.4.6)
6. Observed high CPU for terraform-ls for about two minutes, and then it died down and it's been quiet for about five minutes.

I would love it if someone else tried this and reported their results.

[pincher95]

@jgspratt Did steps 2-4 and observed same behaviour as described in step 6.

[jpogran]

We know this has been a frustrating experience and appreciate your understanding while we gathered feedback and examples to diagnose this issue. We've created #1557 as a central place to see updates on what we are doing to address performance issues in terraform-ls in both the short and long term and pinned it to the repo. We'll be adding more detail there as we implement fixes. As we work on this we'll be recording the content and then closing the individual issues so that everyone has one place to look at instead of searching for individual tickets for updates.

[jpogran]

We've released 2.27.2 which contains two fixes (hashicorp/terraform-ls#1369, hashicorp/terraform-ls#1372) in our continuing efforts to address this issue.

With the fix, we should be back to terraform-ls v0.29.2 levels of CPU usage while maintaining the memory improvements of v0.29.3.

If you have the time, please give it a try and let us know how it works for you. Please open a new issue rather than replying here, as this issue has gotten so many replies with different experiences it is hard to process.

I am going to close this as we're using #1557 as a central tracking issue, so subscribe to that to see continued updates.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.