RDS DBInstance reconcile on masterUserPassword changes

[gecube]

Good day Sirs,

I tried to create a RDS instance:

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: dbserver-5-dev
spec:
  dbSnapshotIdentifier: "****"
  allocatedStorage: 20
  dbInstanceClass: db.r5.large
  dbInstanceIdentifier: dbserver-5-dev
  engine: mysql
  engineVersion: "5.7"
  masterUsername: "root"
  masterUserPassword:
    namespace: infra-dev
    name: dbserver-5-db-password
    key: password
  dbSubnetGroupRef:
    from:
      name: rds-subnet
  publiclyAccessible: false

It was created. Then I tried to add a security group, but I see that it was not added to the instance. I see that the instance is stuck:

  conditions:
    - lastTransitionTime: '2024-05-23T10:41:28Z'
      status: 'True'
      type: ACK.ReferencesResolved
    - message: "InvalidParameterValue: The parameter MasterUserPassword is not a valid password because it is longer than 41 characters.\n\tstatus code: 400, request id: 819086ef-4462-4359-8aa9-f050b3ac3fb6"
      status: 'True'
      type: ACK.Terminal
    - lastTransitionTime: '2024-05-23T10:41:29Z'
      message: Resource not synced
      reason: resource is in terminal condition
      status: 'False'
      type: ACK.ResourceSynced

I recreated the secret and checked that it is precisely 41 symbol. Unfortunately, no changes happened.

I see errors like

{"level":"error","ts":"2024-05-23T10:41:26.972Z","msg":"Reconciler error","controller":"dbinstance","controllerGroup":"rds.services.k8s.aws","controllerKind":"DBInstance","DBInstance":{"name":"dbserver-5-dev","namespace":"infra-dev"},"namespace":"infra-dev","name":"dbserver-5-dev","reconcileID":"d778898e-0331-40e3-a844-6dfd4a2e77a4","error":"the referenced resource is not synced yet. resource:SecurityGroup, namespace:infra-dev, name:limit-rds-to-subnet","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}

in the logs of controller.

Finally I got:

{"level":"info","ts":"2024-05-23T10:41:28.837Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"infra-dev","name":"dbserver-5-dev","account":"178394743802","role":"arn:aws:iam::178394743802:role/ack-ec2-controller","region":"eu-west-2","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","MasterUserPassword"]},"A":"","B":"infra-dev/dbserver-5-db-password.password"},{"Path":{"Parts":["Spec","AutoMinorVersionUpgrade"]},"A":null,"B":true},{"Path":{"Parts":["Spec","BackupRetentionPeriod"]},"A":null,"B":1},{"Path":{"Parts":["Spec","CACertificateIdentifier"]},"A":null,"B":"rds-ca-rsa2048-g1"},{"Path":{"Parts":["Spec","CopyTagsToSnapshot"]},"A":null,"B":false},{"Path":{"Parts":["Spec","DeletionProtection"]},"A":null,"B":false},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"general-public-license"},{"Path":{"Parts":["Spec","MonitoringInterval"]},"A":null,"B":0},{"Path":{"Parts":["Spec","MultiAZ"]},"A":null,"B":false},{"Path":{"Parts":["Spec","PerformanceInsightsEnabled"]},"A":null,"B":false},{"Path":{"Parts":["Spec","PreferredBackupWindow"]},"A":null,"B":"05:01-05:31"},{"Path":{"Parts":["Spec","PreferredMaintenanceWindow"]},"A":null,"B":"fri:23:40-sat:00:10"},{"Path":{"Parts":["Spec","StorageEncrypted"]},"A":null,"B":false},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageType"]},"A":null,"B":"gp2"},{"Path":{"Parts":["Spec","VPCSecurityGroupIDs"]},"A":["sg-06b842635c85a4c5e"],"B":["sg-0e87e0dd9f6d43f31"]}]}

But again - no changes to infra are made.

[gecube]

it helped to restart the controller (!)

[a-hilaly]

@gecube The rds-controller doesn't react on secret data changes. However it should react if the change the secret reference :)

[gecube]

@a-hilaly Hi! Does it documented somewhere?
I thought that I will be able to get the rotation of passwords out of box in such a way. Probably you know ESO - https://external-secrets.io/v0.8.5/api/generator/password/

It could be used for many purposes like getting passwords from external systems like Vault and putting them into k8s as a secrets. But I am using it for generating random password.
The use case is the next. I am spinning up the brand new environment. I don't like to reuse passwords between envs. So I create new password for DB:

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: dbserver-5-db-password
spec:
  length: 41
  digits: 5
  symbols: 5
  symbolCharacters: "-"
  noUpper: false
  allowRepeat: true
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dbserver-5-db-password
spec:
  target:
    name: dbserver-5-db-password
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: Password
          name: dbserver-5-db-password

Then I have a password in a secret named dbserver-5-db-password and I can use it anywhere. But what is more important - it could be refreshed... let's say every week. And I would be happy if ACK could watch the secret and reflect the changes inside of it.