fatal error: concurrent map read and map write

[wagnerone]

Versions

Extension

v2.24.1

VS Code

Version: 1.72.0-insider
Commit: c4722f1bb292284e0437eb7547808a5788fe6430
Date: 2022-09-02T05:31:10.468Z
Electron: 19.0.12
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Darwin arm64 21.6.0
Sandboxed: Yes

Operating System

macOS 12.5 (21G72), Apple M1 Pro

Terraform Version

Terraform v1.2.8
on darwin_arm64

Steps To Reproduce

Expected Behavior

When I save a terraform file, it should be formatted and saved almost instantly.

Actual Behavior

When I save a terraform file, it sometimes is formatted and saved almost instantly.

However, often the saving of the file is delayed by many seconds while terraform language server seems to be processing it.

Sometimes it will format and save. Sometimes it fails and the file is saved without needed terraform formatting changes. This is typically (always?) paired with an error messages related to terraform language server.

Additional context

Our entire team is affected by this problem. We are all on similar hardware and software component versions as well as working with the same (large) set of terraform configurations for our company.

This has been happening to us for a long time, but it has become much more frequent of late and is impacting us to the point of needing to report it.

In the past a restart of Vscode seemed to clear this problem up temporarily, but it always returned after a time. We kind of got used to routinely needing to restart Vscode to clear this issue.

This causes issues with our workflow as we expect the changed file to immediately pick up formatting fixes and be ready for "terraform apply", but find it is not from time-to-time and must wait for it the file to finally be saved before applying.

Lately reloading the window nor relaunching Vscode entirely seem to be a reliable "fix".

[radeksimko]

Hi @wagnerone
Would you mind sharing the log? Especially lines such as these would be very helpful:

2022/09/02 20:47:52 formatting.go:50: Finished 'terraform fmt' in 550.324834ms

Also knowing the approximate size of the file would help (and number of blocks) if you cannot share the whole file as-is.

In most circumstances the formatting should take low tens or hundreds of ms for "reasonably" sized files. If it takes more, then this could be for a number of reasons - resource contention on the OS level, AV or similar. If the process execution is indeed fast, then this could be instead a bug elsewhere.

Thanks.

[wagnerone]

Hi and thank, @radeksimko

Sorry. I didn't get or didn't see a notification for your response. I'll get this information to you soon.

[wagnerone]

More often than not in the past several days I've been getting this.

When I go to iTerm and run terraform format main.tf it reformats the file instantly as expected.

[wagnerone]

When I do a Reload Window and the tf ls starts up and I watch Output->Hashicorp Terraform, I see log entries scrolling for upwards of a few minutes and then the ls issues the above message. It looks like it is acting on every single .tf file in the parent directory tree? Workspace? Not sure about that yet.

I see zero lines that have "terraform fmt" in them at this point while observing the output.

After what looks like 48000+ lines of log entries after restarting the ls by reloading the window, I see this after what looks to be the "normal" lines - i.e., I believe it's the start of the error/crash.

2022-09-09output.txt

[wagnerone]

34K Sep 9 13:50 main.tf

There are ~65 terraform resource blocks in it (if that's what you mean - I'm unsure).

And if run it in iTerm

time terraform fmt main.tf

Reading required version from terraform file
Reading required version from constraint: >= 0.13
Matched version: 1.2.9
Switched terraform to version "1.2.9"

real	0m0.349s
user	0m0.033s
sys	0m0.074s
main.tf

[wagnerone]

It used to just take a long time every now and then when I saved a tf file. I'm having trouble recreating that now as the ls server seems to be crashing before it can do a fmt.

[wagnerone]

I tried creating a new workspace and adding just this terraform repository to it and making edits to the same file.

It still logs 45000+ lines to start with and looks like it walks the entire repository worth of tf files, but it does finish without crashing.

I made 2 edits to the file in question in this new testing vscode window/workspace and see desired speeds here

2022/09/09 14:13:20 formatting.go:50: Finished 'terraform fmt' in 67.025791ms
2022/09/09 14:15:06 formatting.go:50: Finished 'terraform fmt' in 102.252625ms

I have 2 workspaces for terraform work. One is of a single directory containing all of our "live" terraform repositories and the other is of a single directory containing all of our module terraform repositories.

I used to have all terraform repositories in the same workspace, but moved the modules out for organizational reasons a long time ago.

[radeksimko]

Based on the log you shared, it looks like you are running into a bug which we fixed in the most recent release (2.24.2) - Can you try upgrading to that version to see if you can still reproduce the problem?

[wagnerone]

Hi, I have been running 2.24.2 since it was available and this has been happening equally often (if not more?) while running 2.24.2.

[radeksimko]

@wagnerone Can you obtain a more recent log while using the mentioned latest version, so we can better understand the problem? I'm guessing the latest version is no longer crashing, but is still slow for another reason?

Thanks.

[wagnerone]

@radeksimko I'll do that today. It's still crashing even today as I've been working.

[wagnerone]

Sorry for the delay. Here is an error I received today. I've included all the log entries after what looks like normal activity.

2022-09-19output.txt

[radeksimko]

Thank you - this is helpful.

I cannot say with certainty that the bug is related to that feature, but based on the log I would suggest you to try to disable the refcount code lens (which is the default in all recent versions of the extension):

"terraform.codelens.referenceCount": false

It may reduce the chances of running into the bug.

[wagnerone]

Thank you. I will try that next.

The past several days I've had no tf ls, so I spent this morning trying to narrow down what was happening. I moved all my tf repos out of my vscode tf workspace and added them back in small sets reloading the window each time to kick off a ls restart. This helped me identify one repo that seems to be causing the crash consistently in my tests.

I isolated that repo into it's own window/workspace and tf ls crashes consistently there on window reloads.

For reference, in the main workspace with all my tf repos minus the one identified that is causing the ls crashes, it takes 5+ minutes for the tf output to stop scrolling following a window reload. Then, when opening a file or two, it seems to kick off a round of these sort of lines looking at every file. This takes about a minute and completes ok in this workspace.

2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/... 2022/09/21 10:43:12 provider_schema.go:265: PSS: getting provider schema (/Users/user/Documents/work/...

In the window/workspace with the single, offending tf repo, it takes about 2.5 minutes for tf output to stop scrolling on a window reload. Things seem to be OK at this point. However, when I then open a few tf files it logs a ton of lines similar to the above noted lines and crashes during that process.

[wagnerone]

I set "terraform.codelens.referenceCount": false and reloaded the workspace with the offending tf repo only. It took about 2.5 minutes (as before) for tf output to stop after reloading the window. However, this time the ls server did not crash. I opened a few tf files, etc. and it seems stable.

Next I moved the offending repo back into my main workspace with all of my other tf repos, reloaded the window, waited for tf output to stop scrolling (about 7 minutes) and began to open files.

Things seems stable in this case now too. Woo!!

[wagnerone]

Oh shoot. :( I spoke too soon. The ls crashed after bopping around a few files in the combined workspace with "terraform.codelens.referenceCount": false set.

Here is the tf output at crash time
2022-09-21output.txt

[wagnerone]

Looking for an update on this, please. Even having the workspaces/repositories separated as I describe I still get ls crashes occasionally. It's substantially less often, thankfully, but it's inconvenient to have to work in related repositories spread across multiple workspaces. Thank you!

[radeksimko]

After some experimenting I came up with a failing test which discovered a race condition in nearly the same place/context: https://gist.github.com/radeksimko/6430e0e7e135bb11b0e2176dd0c3989c

Still need to dig deeper to find the root cause and solution though.

Thanks for the patience!

[radeksimko]

I believe I found the root cause and have a fix in #1086 / hashicorp/terraform-schema#149 (currently pending review)

[wagnerone]

Great! Thank you very much!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.