LS state & performance refactoring

[dbanck]

• Depends on: Refactor SchemaMerger interface terraform-schema#335

---

TL;DR: This PR tries to solve two main problems:

• slow startup performance & high resource usage for larger workspaces
• not easy extendable internal state for new languages, like Terraform Test & Stacks

---

I would suggest that you review this PR commit by commit. First, I introduce an event bus to tie everything together. Then I introduce the concept of features to the codebase along with three new features:

• A feature for root modules (.terraform, .terraform.lock.hcl)
• A feature for Terraform files (*.tf, *.tf.json)
• A feature for Terraform variable definition files (*.tfvars, *.tfvars.json)
  Each feature decouples and isolates all functionality related to handling these files (or directories). It has its own internal state, its own set of jobs, its own subscription to specific event topics, and its own decoder (if applicable).
  It's intended to have more features in the future: a stacks feature, a tests feature, and some for other Terraform-related languages/things.

Now that the features are in place, we can remove everything from the existing packages that belongs in a feature. Mainly we can clean up the global state, the terraform package and the walker & indexer. Only structures that are relevant to the language server as a whole (job scheduler, schema store) or that are intended to be used in conjunction with multiple features (change batches) should live at the global package level.

I also refactored the walker and removed the scheduling of jobs when walking a workspace after opening it. We still detect and collect all directories relevant to terraform-ls, but we don't index them. Only when a user opens a file do we schedule jobs to parse the modules, variables, and root module files if applicable. This results in faster startup times and overall less CPU and memory usage.

Also, the text synchronization handlers now don't block on incoming requests. Previously, we would wait for all jobs to finish processing before accepting the next didChange request. Now, instead, we wait in the language feature handlers (e.g. complete or hover) until all jobs for a directory have been processed before returning a result. This should result in a better user experience and better job deduplication in the queue.

---

Known issues:

Missing decoding of installed modules fixed in #1760
I plan to address this in a separate PR. Since we don't index the whole .terraform directory at startup, we need to introduce another way to detect installed modules. We can extend the existing local module detection on open to detect installed modules as well. The only missing piece is a mapping from a module's source to its installation location on disk.

The global workspace symbol LSP method is now less accurate
Since we only keep open modules in memory, the handler will only report symbols for all modules that have been opened at least once. This is a tradeoff for not indexing the entire workspace at startup. We can still look at a low resource background index job or kick off indexing the workspace when a sends a workspace symbol request. At that point, we can report progress in the UI.

Flaky tests?
Keep monitoring

---

Rough performance estimates:
A a workspace with ~50k lines of Terraform code
B a workspace with ~600 lines of Terraform code
on an Intel i9 2,4Ghz 8 Core MacBook Pro

main

A: 19,09s 229MB (startup, unitialized)
A: 23,38s 282MB (3 open modules, unitialized)

B: 0,09s 11,9MB (startup, unitialized)
B: 0,23s 12,3MB (2 open modules, unitialized)

new

A: 0,14s 9MB (startup, unitialized)
A: 5,26s 204MB (3 open modules, unitialized)

B: 0,02s 6MB (startup, unitialized)
B: 0,11s 11,3MB (2 open modules, unitialized)

---

Closes:

• Improve language extensibility #1614
• Split monolithic state into multiple entities #1626
• Create a generic abstraction over all state entities #1627
• Reduce the amount of initial work when walking a workspace #1629
• Only schedule jobs relevant to the files when indexing a directory #1630

[jpogran]

[ansgarm]

Do we want to create issues for this somewhere? (I'm fine with leaving these comments as is)

[ansgarm]

[ansgarm]

I just noticed that we might need to update our docs (e.g. docs/architecture.md) in a follow-up PR to reflect the refactoring.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.