internal/state: avoid deadlock caused by mid-flight job change

[radeksimko]

There is a relatively rare case of when we may end up processing the same job twice here:

terraform-ls/internal/state/jobs.go

Lines 188 to 211 in bff4e90

	func (js *JobStore) awaitNextJob(ctx context.Context, openDir bool) (job.ID, job.Job, error) {
	txn := js.db.Txn(false)
	wCh, obj, err := txn.FirstWatch(js.tableName, "is_dir_open_state", openDir, StateQueued)
	if err != nil {
	return "", job.Job{}, err
	}
	if obj == nil {
	select {
	case <-wCh:
	case <-ctx.Done():
	return "", job.Job{}, ctx.Err()
	}
	return js.awaitNextJob(ctx, openDir)
	}
	sJob := obj.(*ScheduledJob)
	err = js.markJobAsRunning(sJob)
	if err != nil {
	return "", job.Job{}, err
	}

1. a job e.g. ID 42 is found by e.g. "closed scheduler" (*)
2. in the split second between finding this job and marking it as running (which itself is correctly guarded by an internal memdb mutex), the involved directory is open.
3. the same job ID 42 is then also found by "open scheduler" (**)
4. whichever scheduler obtains the memdb mutex first marks that job as running
5. the other scheduler then waits for the mutex and attempts to do the same
6. because that fails, we return error, making the scheduler to effectively shut down, practically causing a deadlock in places which wait for jobs to finish - as they'd never finish

(*) - the scheduler responsible for processing jobs involving directories which are not open
(**) - the scheduler responsible for processing jobs involving directories which are open

While this may be rare, I ran into that bug myself this morning, as can be seen from this log snippet:
https://gist.github.com/radeksimko/7ce4e9a7d0c8bbc65bd65e277dc3bd92

[dbanck]

Nice catch and excellent test case!

[github-actions]

This functionality has been released in v0.26.0 of the language server.
If you use the official Terraform VS Code extension, it will prompt you to upgrade to this version automatically upon next launch or within the next 24 hours.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.