K8SSAND-1615 ⁃ Modular Secret Backend Epic

[adejanovski]

Background

K8ssandra v1.x and K8ssandra-operator only work with Kubernetes secrets as secret provider. This prevents companies using K8ssandra if their security policy doesn’t allow usage of Kubernetes secrets.
We would need to provide an API that has the following characteristics:

• Secrets should be injectable into pods directly, without requiring access from the operators
• Secrets providers should be pluggable without requiring implementation specific code in K8ssandra-operator or cass-operator (generic external provider)
• It should be possible to add providers at runtime
• Secrets should be mounted in a common way so that the operator can perform validation checks using a validation webhook
• Management of default secrets should be disabled when using an external provider

Design

This proposal leverages dynamic admission control to allow extension at runtime, without recompiling or redeploying K8ssandra-operator.

This design is widely inspired by the Hashicorp Vault agent which uses the same technique for secrets injection at runtime.

Tasks

Beta Give feedback

1. K8SSAND-1616 ⁃ Introduce a secrets provider setting in the CRD #599
   +0
2. K8SSAND-1617 ⁃ Disable secrets management and replication with the external secrets provider #600
   review +1
3. K8SSAND-1624 ⁃ Design the required annotations for the internal secrets provider #606
   done +1
4. K8SSAND-1430 ⁃ Allow configuring annotations and labels on services, statefulsets, deployments and pods #501
   done +1
   
5. Replace secrets refs for the Cassandra pods with the mutating webhook #601
   0 of 1
   discuss done +2
6. Replace secrets refs for the Reaper pods with the mutating webhook #602
   +0
7. Replace secrets refs for Medusa with the mutating webhook #603
   +0
8. K8SSAND-1632 ⁃ Allow Medusa to take CQL credentials through a file #604
   +0
9. K8SSAND-1623 ⁃ Create a mutating webhook for the internal secrets provider #605
   done +1
10. K8SSAND-1625 ⁃ Use a job to register the cluster into Reaper #607
    +0
11. Create an image for the secrets injection init container #608
    done +1
12. Allow Reaper to read JMX/Cassandra/UI credentials from a file instead of env variables thelastpickle/cassandra-reaper#1210
    blocked new +2
13. Allow Medusa to take CQL credentials through a file thelastpickle/cassandra-medusa#493
    done new +2
    
14. Ability to read required secrets from other sources (file) than Kubernetes cass-operator#520
    enhancement product-backlog +2
15. Ability to read required secrets from other sources (file) than Kubernetes #964
    product-backlog +1
16. Allow specifying a list of containers in which the webhook will mount the secrets #965
    3 of 3
    done +1

┆Issue is synchronized with this Jira Story by Unito