Deleted variables from API remain in configmap

[shreddedbacon]

Describe the bug

When deleting an environment variable from the API, it doesn't remove it from the lagoon-env configmap. This is a problem when using the existence of the variable to enable/disable a feature. Like BASIC_AUTH_USERNAME and BASIC_AUTH_PASSWORD.

To Reproduce

Steps to reproduce the behavior:

1. Add global|runtime variable to the API
2. Deploy environment
3. See variable is added to the config map
4. Delete variable from the API
5. Deploy environment
6. See that deleted variable is still in the config map

Expected behavior

Deleted API variables should be removed from the configmap

Impacted customers

At least 2 large customers in APJ are impacted by this.

[shreddedbacon]

It should be possible to directly(or via msgqueue) patch the configmap when adding or deleting global|runtime variables directly via the API?

[seanhamlin]

I like your idea @shreddedbacon, because then customers can still manually manipulate the configmap (if needed), and we can still delete environment variables.

[seanhamlin]

This issue is impacting another large customer in APJ.

[chandanatk]

As reported by a customer, if a user tries deleting and recreating environments as a workaround for this issue, it exposes another bug. When an environment is deleted, the environment variables are still seen with lagoon cli, but when a new deploy recreates the environment, variables are actually deleted. The user then needs to recreate the variables in time for the deployment that is already running.

Steps to reproduce the behaviour:

1. Delete environment
2. Check the variables using Lagoon CLI and see they exist
3. Deploy the environment
4. Again check the variables using Lagoon CLI and see they don't exist
5. Add the variables

[shreddedbacon]

As reported by a customer, if a user tries deleting and recreating environments as a workaround for this issue, it exposes another bug. When an environment is deleted, the environment variables are still seen with lagoon cli, but when a new deploy recreates the environment, variables are actually deleted. The user then needs to recreate the variables in time for the deployment that is already running.

Steps to reproduce the behaviour:

1. Delete environment
2. Check the variables using Lagoon CLI and see they exist
3. Deploy the environment
4. Again check the variables using Lagoon CLI and see they don't exist
5. Add the variables

This is a funky one.
When deleting an environment, the environment remains in the API still (its still accessible from a historical data perspective) so the variables attached to that environment will remain available.

But as it has been deleted, when a new deployment starts, it will create an entirely new environment in the API making the old one no longer accessible by the same name (only by ID), and this will have no variables attached to it, even though the old one did.

I'm not sure if this is related to this issue, or if its better raised as a new issue entirely, where we support adding variables to environments, before the environment is created. Or being able to pre-seed an environment, then deploy it afterwards.

[grahamethompson]

As a customer we're obviously experiencing this issue. Would be great to have this resolved - thanks, much appreciated!

[shreddedbacon]

As we were discussing this, https://github.com/uselagoon/lagoon/pull/2348/files#diff-e77377f4b63b4a8a4d877d7a1adb7aa9cab7252db63dd9c3f43b64d5dc972408R1073 with the new implementation of this PR, when the op-remove runs on the api-only variable configmap, we could also do an op-remove on the existing lagoon-env configmap

[shreddedbacon]

@rocketeerbkw have you had a chance to work on this?