Slow formatting due to module 404s

[lingrino]

Server Version

❯ terraform-ls version
0.30.2+Homebrew
platform: darwin/arm64
go: go1.19.6
compiler: gc

Terraform Version

❯ terraform -version
Terraform v1.3.9
on darwin_arm64

Client Version

VS Code:
Version: 1.75.1
Commit: 441438abd1ac652551dbe4d408dfcec8a499b8bf
Date: 2023-02-08T21:34:01.965Z (1 wk ago)
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: 10.2.154.23-electron.0
OS: Darwin arm64 22.3.0
Sandboxed: No

Terraform Configuration Files

# meta.tf
terraform {
  cloud {
    organization = "<org-name>"

    workspaces {
      name = "aws-main-s3"
    }
  }

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

provider "aws" {
  region = "us-west-2"

  default_tags {
    tags = {
      workspace = "aws-main-s3"
    }
  }
}

# s3.tf
module "module_name" {
  source  = "app.terraform.io/<org-name>/s3/aws"
  version = ">=2.0.0"

  env  = "redacted"
  name = "redacted"
}

# repeat above module call 23 more times

Log Output

Encrypted log file: https://drive.google.com/file/d/1WtANFqHCI6CpcRFbtIpvps4-JRmJwX5A/view?usp=sharing

Expected Behavior

When I save a file it should be formatted quickly

Actual Behavior

Saving a .tf file takes 10+ seconds depending on how many modules are in the workspace. Even if this is a save with 0 changes in an empty test.tf file.

Steps to Reproduce

1. Open vscode the root of our repository
2. Open any .tf file in a larger (more than 3 modules?) workspace
3. Make any change
4. Attempt to save the file

More details

• We use terraform cloud for remote state but not remote applied. All of our modules are in separate repos and loaded through our terraform cloud private registry.
• We have a repo called infra-terraform with multiple nested workspaces. Workspace directories are named terraform/<provider>/<service>. Usually I have the entire repo opened in vscode but sometimes only one of the nested workspaces. We change and apply terraform in each of these nested workspaces. A simplified example:

terraform/
  aws/
    s3/
      main.tf
    cloudfront/
      main.tf
  gcp/
    dns/
      main.tf

What I've Tried

This is an issue I've been having for months so I don't think it's a recent change. I just got around today to filing this bug because it's also experienced by others on my team. Here's some things I've already tried:

• I've tried all combinations of recommended formatting settings in the extension readme here. Originally I thought this was my problem but none of these recommendations work.
• I use a custom location for .terraformrc and I use the plugin cache. I've restored both of these to the defaults but still have the issue.
• I've deleted the .terraform directory in all my workspaces and rerun terraform init
• I thought this could be an issue with our directory structure. But I've tried moving one of our nested workspaces into its own empty root folder and still experience the issue. This also happens even if I open one workspace instead of the whole repo.
• My API key is valid. I can terraform init/plan/apply without any issue. I have no problem access the referenced modules.
• I've search all issues in this repo and the extension repo but I don't see any similar open issues.

Log Output

From the log output it looks to me like terraform-ls is making some remote network calls looking for our modules and for some reason it's getting a 404. The network calls appear to block the usual formatting and other extension features that I expect. This prevents me from saving files. Sometimes this resolves after a few seconds and sometimes it times out completely after ~30 seconds. The experience is worse in our large workspaces with 20+ module calls.

In the logs I see that terraform-ls is frequently running jobs named OpTypeGetModuleDataFromRegistry. I see in the code that this function eventually calls GetModuleData and based on the errors I think this is what's failing:

https://github.com/hashicorp/terraform-ls/blob/main/internal/registry/module.go#L62-L85

Whenever I open a workspace (for example our s3 workspace in the example above) I regularly see the following error in the logs where the error count is equal to the number of module calls in the workspace. For example in our S3 workspace we call 24 modules (different buckets) and I see the following error 24 times:

2023/02/17 17:39:43 jobs.go:383: JOBS: Finishing job "2338": "OpTypeGetModuleDataFromRegistry" for {"file:///Users/<dirname>/projects/work/infra-terraform/terraform/aws/main/s3"} (err = 24 errors occurred:
	* unexpected response 404 Not Found: {"errors":["Not Found"]}

[22 more of these]

	* unexpected response 404 Not Found: {"errors":["Not Found"]}


, deferredJobs: [])

I hope this is enough information to pinpoint the issue. Thank you!

[aclarknexient]

EDIT: commenting out the module source attribute will fix the slowdown. The "Extension Host" output from vscode says Error: main.tf (34,5): unknown attribute "name" when hovering over a given attribute of a module, in this case "name". So perhaps we need an option for the terraform language server to not try to fetch module details?

EDIT: The slow format-on-save behaviour is still happening with the preview extension. This is super annoying!!

---

This is also happening to me with the latest stable version of the VSCode extension. Upgrading to the preview version of the extension appears to have solved it. I still get the 404s but they don't seem to affect the formatting/saving speed.

Preview extension name is "HashiCorp.terraform-preview"

I'd like to see what URLs the language server is attempting to fetch, so I can try to replicate this via the command line or see whether the URL is incorrect.

[aclarknexient]

When I change the module source to a known good URL unrelated to modules, the 404 errors go away (obviously!) and the slowdown disappears.

[dbanck]

Thank you for raising the thorough bug report!

We released the new VS Code extension 2.26.0 version (and 0.31.1 language server version) with a potential fix. So please try it and let us know if it solves this issue.

[lingrino]

@dbanck this is much better! Thank you!

I still have the same slow experience the first time I try to save a file but afterwards it is much better. Given the caching fix it seems like that is expected? Is there anything I can do to resolve these 404s on my end?

Thank you again!

[radeksimko]

Is there anything I can do to resolve these 404s on my end?

Assuming this is all related to the private Registry, i.e. modules with

source  = "app.terraform.io/<org-name>/..."

and/or you're sourcing submodules (involving //), then there's nothing to do on your end at this point.

We intend to address both as part of

• Support uninitialised submodules in the Registry #1035
• Support uninitialised modules from a private Registry #1270

We are also hoping to eventually improve the performance in this area as part of #958

I hope that helps.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.