[BUG] - Prevent SMTP realm settings from being wiped by terraform

[costrouc]

Describe the bug

Currently it is not possible to set keycloak realm settings without a redeploy wiping settings.

Expected behavior

Don't wipe keycloak user set realm settings

OS and architecture in which you are running Nebari

linux

How to Reproduce the problem?

Redeploy nebari

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

[kenafoster]

@costrouc @dcmcand Here's what I've found and recommendations:

TL;DR - short term solution to ignore_changes to the entire keycloak_realm is probably the way to go (and safer than it sounds). Might need to revisit when we talk about applying optional higher security configurations

The necessary realm settings get wiped because they're managed by the keycloak_realm release in the TF provider https://registry.terraform.io/providers/mrparkers/keycloak/latest/docs/resources/realm

• so if they're not specified in the TF object, they'll get wiped to null/default every time TF re-applies. https://github.com/nebari-dev/nebari/blob/develop/src/_nebari/stages/kubernetes_keycloak_configuration/template/main.tf

FYI in addition to the SMTP settings block, there's also a verify_email flag that needs to be set to true. (Nebari doesn't have an option for this either, which makes sense... if SMTP settings can't be reliably configured, then requiring users to verify email (via an outgoing Keycloak email, of course) is a dead end

Here's some options. My recommendation for the <2 week release is option 1 since easiest but open to discuss

1. Add a lifecycle-ignore_changes block to the keycloak_realm resource

DECISIONS:

• Do we just ignore the specific SMTP config and verify email blocks, every property that isn't configurable directly via Nebari, or even ignore ALL changes including those deployed via Nebari?

THOUGHTS:

• Ignoring "every unconfigured property" will need to be updated to add new properties as the TF provider changes
• We already manage users, groups, etc. in Keycloak directly and not Terraform, so I think its a natural extension of that process to make the Keycloak realm "user managed" in the Keycloak console as well
• Also, looking thru the code as it is today, everything is hard coded for that object (except realm_display_name for some reason?)
• I think (need to confirm) that even if 'ignore_changes' is set on a property, that TF will still pick up if any changes in the code since the last apply - so for example, if a future Nebari release changes a property, TF will still pick that up and apply
• One catch here is that if we do a "high-security" flag in future (such as making an entire Neb platform apply NIST controls or whatever), then there will be some security settings applied at the realm user that we want to make sure can NOT be overridden (permanently) outside of Terraform... in this scenario we would want TF to not ignore any changes that remove security and reapply them with each deploy

2. Additional settings via Nebari Config?

DECISIONS:

• Do we extend the inputs field-by-field, or do we do a freeform keycloak_realm_overrides that just applies whatever they put as attributes of the keycloak_realm object? Or some hybrid, such as free-forming only within certain blocks?(smtp_server, security_defenses, etc... if you look at how the TF object is set up, this isn't as dumb as it sounds at first)

• How do we handle sensitive info like SMTP password? One solution would be to ignore changes just on password field and instruct/hope the user puts a temp or placeholder in there when Nebari runs (rotating the keycloak root user initial password works kind of like this). A more comprehensive way would be to create an outside secret (e.g. AWS Secret Manager) in an earlier Nebari stage and the deploy stops and asks the user to populate it (kind of like how Nebari deploy handles manual DNS) before continuing the deployment and reading the secret value from a TF data object in the Keycloak stage. But that requires a lot of setup to get right (including making the password secret conditional) and also will be a headache to make compatible with all clouds (since the kubernetes_keycloak_configuration stage is currently provider agnostic)

  THOUGHTS:

• Given that nothing else having to do with the Keycloak realm is customizable config via Nebari anyway right now, this doesn't seem mandatory

• However, we also have the problem that if we accidentally blow away Keycloak's data, we lose any settings that aren't in the Nebari config. (But this is a problem probably better handled by Keycloak backup anyway... given the blowing away Keycloak DB would also lose all the users, etc.)

• If we also want to apply the same settings across multiple Nebari platforms (i.e. configure some custom security settings in a "dev" environment then promote to production after testing)... promoting (and maintaining) custom configs will be easier with more extensibility

[sblair-metrostar]

Would be nice if whatever solution comes of this isn't exclusive to the SMTP settings. There are several realm level settings I've tried to adjust in the past that have this same issue, such as brute force detection, various login options, token expiration, etc.... Realm themes seem to be the only thing I've tried that survive a deployment after having been set manually and I can't tell if that's a feature or a bug of the Terraform provider.