fix: Prevent parsing invalid tokens

[radeksimko]

Fixes #217

The user will often produce invalid/incomplete config as they type, so diagnostics are ignored - which is by design in case of the parser. However the parser also assumes that the sequence of tokens it receives from the lexer is valid, i.e. that the important check for invalid tokens is not ignored.

There are some configs which one might expect to be "parseable", e.g.

provider "null" {
}

variable "name" {
  default = "
}

but the parser isn't ready to deal with such config yet and the lexer returns the trailing } as string literal.

If we wanted to perform completion near the opening quote we would be completing in the context of a string literal. We don't do RHS completion yet anyway, so it is not a big deal to just error out now, but I reckon we will need to revisit this to e.g. support the following

provider "null" {
}

variable "name" {
  default = "${var.
}

We could virtually append closing quote there, to make it a valid sequence, but the question is when (in terms of codepath). If that has to be done before we send bytes to the lexer, then we are practically reimplementing the lexer. We could also just check for particular diagnostics and rerun the lexer - which might be fragile, but it could be guarded by tests, so probably fine.

[aeschright]

LGTM

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the context necessary to investigate further.