Indexing seems to block completion

[lanox]

Hi

I am not sure if I am asking/reporting this as a bug or more of a question.

I have noticed that my lsp.log from neovim is going crazy every time I use terraformls lsp server, it literally crates 100s of line of [ERROR] messages.

I have attached image just for reference as logs contain paths which I was not comfortable to expose, but if needed I could clean it up and post it in here.

My setup is simple.

I use
nvim version: NVIM v0.9.0-dev-1343-gbcd25b200-dirtywith mason plugin for installing LSP servers.

terraform-ls version
0.29.3
platform: darwin/arm64
go: go1.19.1
compiler: gc

I have also noticed when I open the terraform files it actually takes few seconds for LSP server to start working.

I am not sure what else I can provide but if you need anything else please let me know.

Thanks

[queue-tip]

I get the same results with Neovim 0.8.1 and terraform-ls 0.30.1.

In addition, I've extended the timeout for formatting to 5000 ms and still get a timeout every time. Formatting used to work relatively well.

[radeksimko]

Hi @lanox

On the note of logs:

As Martin explained already in https://discuss.hashicorp.com/t/terraform-lsp-log-going-crazy/47715/4 the [ERROR] prefix in NeoVim appears to be just an indication of the output coming from stderr.

While logging to stderr isn't entirely unusual in Unix world, some other language servers do not send logs by default and/or utilize the LSP method window/logMessage. This is an improvement we could consider implementing! It however doesn't mean that we can eliminate all logs, as there will always be details which relate to launching the server. We could reduce it to minimum though.

This will also require us to potentially reflect these changes in https://github.com/hashicorp/vscode-terraform to ensure that logs remain easily available in the right place.

---

@lanox @queue-tip To find the reason behind the slowness, we'd need to take a look at your whole log. Also, if you could come up with a minimal repro case with a directory with only one *.tf file and no nested directories, that would be very useful!

There are possibly two reasons this could be happening in theory:

1. You use a wrapper around terraform which causes some delays in executing Terraform CLI, or there is AV or another endpoint protection software which is delaying the execution. I experience the latter myself with CrowdStrike Falcon recently, although I wasn't able to come up with a reliable repro.
   • You could reproduce this by running cat file.tf | terraform fmt - (or time terraform fmt file.tf) which is more or less equivalent of what the LS does when formatting a file. Formatting a small file should take low tens of ms and even average-sized files should generally be processed well within a second.
2. Firewall or other endpoint protection SW blocking/slowing external requests, or simply slow connection to registry.terraform.io, see Slow jobs block LS/RPC requests #1063 where we track this bug specifically.
   • You could try to remove any module blocks with external source addresses from the config file to see if the problem goes away.

If you are worried about the data in the log, feel free to send them privately to radek <at> hashicorp.com and/or encrypt it using my key, as available at https://keybase.io/radeksimko

[nocode99]

I am also experiencing this error on Neovim 0.8.1 with terraform-ls 0.30.1 (also on 0.29.0-dev). I've attached a lsp.log file sample. Here's what I did:

1. Deleted contents in lsp.log
2. Started writing a main.tf in a new directory while tailing lsp.log
3. No events started. After I did something simple like resource "aws_instance" "test" {} and then ran terraform init
4. Re-opened main.tf and started seeing a lot of errors spew out.
5. When I opened up autocompletion, as I started going through each option, lots of errors were getting generated as I was scrolling. If I went back to a previous parameter, no errors were generated.
   
6. Once I added a single parameter, errors were being generated consistently though not at an alarming rate.

I noticed an alarming high errors when I initialized one of our modules. We use a mono repo for our modules and it seems it's traversing through all the modules and throwing errors there. I've attached a sample where nvim opened a tf file for around a minute.

lsp.log

[nocode99]

I should add that this is happening through LunarVim which handles the LSP integration. On my own custom neovim setting, this doesn't happen. I'll open an issue with LunarVim

[radeksimko]

@nocode99 I do not see any actual error, rather log lines prepended with [ERROR], which as I tried to explain above is because all logs are sent to STDERR, which in turn NeoVim (or the relevant plugin) interprets as errors.

The screenshot also suggests that completion is working as expected. Is there anything in particular not working as expected, or were you just referring to the verbosity of the logs?

[nocode99]

@radeksimko apologies - as I missed that. I've been doing some more troubleshooting. I was testing out LunarVim and could not figure out a way to disable those "Error" logs from generating.

I went back to my own nvim and did notice that 0.28.1 performs better than 0.30.1. In a directory that calls our mono-repo module and some public modules, the logs generated in CocCommand workspace.showOutput languageserver.terraform-ls is a lot more. I took a screencast:

1. terraform-ls 0.28.1 you'll see the options loading faster and note the # of log lines
2. Switch to 0.30.1 and you'll see tons of logs being generated and waiting a long time for the LSP options to load

Screencast

Logs

terraform-ls_0.28.1.txt
terraform-ls_0.30.1.txt

[radeksimko]

Thank you for those details @nocode99

It is likely that the amount of logs is correlated with the number of modules that you have in your workspace (monorepo, as you say). The language server began to index all modules, rather than just the initialised ones in more recent versions (0.29.0) as per #724 which would explain the difference.

That said, the indexing should be done on the background and should not affect the completion - so that sounds like a bug, although it is likely it's the same one as filed under #1063

I will keep this open at least until we address #1063

[radeksimko]

Hi @nocode99,
We released 0.31.1 which includes a performance improvement that should help address the issue related to inaccessible module data, which may be because the modules come from the private Registry (e.g. app.terraform.io/<org-name>/... and/or involve submodules (//).

We intend to eventually address both as part of

• Support uninitialised submodules in the Registry #1035
• Support uninitialised modules from a private Registry #1270

We are also hoping to eventually improve the performance in this area as part of #958 and also #1056

Finally we intend to address the logging issue as part of #1271

If you believe the root cause of your issues is not covered in any of these linked issues I'd encourage you to file a new one and provide some details, otherwise we appreciate the patience until we address the above issues.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.