Enable password-protected Redis for HA

[crenshaw-dev]

Summary

Our HA manifests should support running Redis with password protection. The redis-haproxy config doesn't currently support auth.

Motivation

Password-protecting Redis is #3130. Adding password protection provides an additional layer of security to protect cached information (e.g. rendered manifests) from unauthorized access.

Proposal

Modify the redis-haproxy manifests to support passing auth information.

[edmondshtogu]

Is there any plan for adding this functionality any time soon?

[crenshaw-dev]

@edmondshtogu in my mind, yes. 😄 I'm not precisely sure when I/someone will get to this though.

[jdoylei]

@crenshaw-dev - Thanks for capturing this issue. We're running the HA Argo CD install, and our security team flagged the unprotected Redis as an issue. If I understand it right, the only way right now to password-protect Argo CD Redis is to replace the HA install with the simple install, right?

[crenshaw-dev]

@jdoylei based on this comment, you might be able to modify your install manifests to get it to work with HA: #3130 (comment)

[jdoylei]

@crenshaw-dev - Thanks for the tip! If I translate that comment into changes applied to Argo CD's HA namespace-install.yaml, it looks like there are 4 "tcp-check send PING" commands in the haproxy.cfg configmap entry. Would I need to replace each of the 4 occurrences, per that comment?

[jessesuen]

Michael asked me to jot down some notes about redis gotchas with respect to password updates in redis-ha. Theres some challenges if you ever need to update ACL on a ha redis:

Redis (or at least the way we use it) does not replicate ACL configuration to the other two nodes (unlike the actual redis data which does replicate). So if you are connecting to the shared hostname (e.g. redis-cli -h argocd-redis-ha-haproxy:6379) and running acl commands, you will only be updating the ACL configuration for one of three redis servers (whichever is the master).

Redis does have an ACL LOAD command to reload an ACL file from disk after it's changed, but because kubernetes takes time to propagate secrets to volume mounts, you don't know for certain when that acl file was updated (it can take minutes). You also still have the problem of needing to run ACL LOAD on all three nodes.

So, the easiest way to update redis ACLs after a change, is to bounce all three redis servers after changing the acl on disk (but restarting redis ha statefulset can be slow, and can sometimes cause issues).

Note that all of the above is only ever a problem if ACL needs to be changed or password needs to be rotated on an already running redis.

[jdoylei]

@jessesuen and @crenshaw-dev - Just revisiting this, because we had some other reason to override haproxy.cfg (see #15319), and I had some second thoughts about the haproxy.cfg tcp-check send AUTH command shown in #3130 (comment). I was trying to decide whether there was some way to extract the password from haproxy.cfg, since embedding the password seemed to create more of a security issue. Would you expect that the haproxy.cfg file specified in Argo CD's argocd-redis-ha-configmap should be able to reference Environment Variables, like they're documented by HAProxy?

[todaywasawesome]

Closed by f1a449e