[ENH] - Prevent unsupported changes to the nebari-config file

[Adam-D-Lewis]

Feature description

In the nebari config file, there are some values which should only be set for the initial deployment, and there are some values that can be changed even after the initial deployment without ill effect.

Examples of values which should probably only be set once are "project_name", "provider", "terraform_state", and some of the "storage" section settings on some of the cloud providers.

Those values can be changed, but will usually result in an undesired and unexpected change during the next deployment. For example, if you change the project name, your entire cluster will be destroyed and a new cluster will be created. If provider is changed, then a similar destruction and recreation occurs. If you change terraform_state from remote to local, I believe you get an error message that we haven't documented how we users should resolve it. As for storage, I believe on some of the cloud providers, if the storage sizes are changed, the disk is deleted and a new one is created. Anyway, you get the idea.

It would improve user experience if we disallow changes to certain values and display a helpful error message. This is not possible currently b/c we can't detect when a value changes b/c we don't track changes to the nebari-config file in nebari. I propose we start doing so. For the examples I've listed above we would only need to store the values from the prior applied nebari-config.

The way I propose we do this is by storing the prior applied nebari config file in the terraform state (e.g. via a terraform_data resource). We can then read it, note the changes and raise warnings or errors as desired if there were any disallowed changes in a new stage between the current stage 1 and stage 2. There may be some edge cases to think through, but that is the general idea.

Ideally, the checks for disallowed changes should be stored in the code for each stage so extensions can also specify what is allowed/disallowed. Then we run all those checks on this new stage 1.5 (for lack of a better name atm).

Value and/or benefit

Better User Experience

Anything else?

No response

[viniciusdc]

Might be worth as context, but we already save the config file as a secret as part of the helm extension stage (if I am not mistaken), also it's kept in a /temp location when render is called.

nebari/src/_nebari/stages/nebari_tf_extensions/template/nebari-config.tf

Lines 1 to 10 in bb54646

	resource "kubernetes_secret" "nebari_yaml_secret" {
	metadata {
	name = "nebari-config-yaml"
	namespace = var.environment
	}
	data = {
	"nebari-config.yaml" = yamlencode(var.nebari_config_yaml)
	}
	}

[Adam-D-Lewis]

also it's kept in a /temp location when render is called.

@viniciusdc Could you point to the part of the code where that happens? I've only seen the nebari-config read from file and kept in memory after that.

[Adam-D-Lewis]

Might be worth as context, but we already save the config file as a secret as part of the helm extension stage

@viniciusdc Do you know if that secret is used for any purpose currently?