CARM ack-role-account-map ConfigMap updates are not propagated to ACK controllers

[itaiatu]

Describe the bug
CARM ack-role-account-map ConfigMap updates are not propagated to ACK controllers.
Maybe related to #2011.

Steps to reproduce
I performed the following scenario:

• CARM ack-role-account-map ConfigMap

apiVersion: v1
data:
  "00000000000": arn:aws:iam::00000000000:role/X-ack-iam-controller
  "11111111111": arn:aws:iam::11111111111:role/X-ack-iam-controller-assumedrole
kind: ConfigMap
metadata:
  name: ack-role-account-map
  namespace: ack-iam-controller

• Changed the value of 11111111111 in the data field in the to

"11111111111": arn:aws:iam::11111111111:role/X-ack-iam-controller-assumedrole2

• IAM controller detected the change

{"level":"debug","ts":"2024-06-13T09:56:32.974Z","logger":"cache.account","msg":"updated account config map","name":"ack-role-account-map"}

• After changing the value of "11111111111" AWS subscription id to a dummy value (adding the 2 at the end of the value), I still managed to create a Role and a Policy in the ns-ack-test namespace (see below the Namespace spec). This is not the desired behaviour, since that role doesn't exist (with the 2 character at the end of the string).

• After I restarted the IAM controller's pod, got 403s, the expected behaviour, since that role is not a valid one

{"level":"debug","ts":"2024-06-13T10:54:38.873Z","logger":"ackrt","msg":"<<< rm.sdkFind","kind":"Policy","namespace":"ns-ack-test","name":"ack-test-policy","account":"11111111111","role":"arn:aws:iam::11111111111:role/X-ack-iam-controller-assumedrolee","region":"us-west-2","is_adopted":false,"generation":1,"error":"AccessDenied: User: arn:aws:sts::00000000000:assumed-role/X-ack-iam-controller/1718276077280567570 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::11111111111:role/X-ack-iam-controller-assumedrolee\n\tstatus code: 403, request id: ad4f99b2-5b84-403e-ac16-ee1e56293ef0"}

• Vice-versa, if I start with a CARM wrong configuration (wrong assumedrole names), I get 403s as expected, and after I fix the role name in ack-role-account-map to match the correct assumedrole, I still receive 403s when trying to create resources.

• This makes me think that although the log messages say that the runtime cache is updated, the change is not propagated to ACK controllers.

Expected outcome
When ack-role-account-map is edited, ACK controllers will use the updated values from the data field.

Environment

• Tested with the latest version of IAM controller (v1.3.8 that contains the updated v0.34.0 runtime version, which also integrated this PR Resolve race condition between CARM ConfigMap and reconciler for annotated namespaces runtime#138)

• The namespace where I tried to create a Role and a Policy with the IAM controller

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    services.k8s.aws/owner-account-id: "11111111111"
  name: ns-ack-test
spec:
  finalizers:
    - kubernetes

• Kubernetes version: 1.28
• Using EKS (yes/no), if so version? 1.28
• AWS service targeted (S3, RDS, etc.): Multiple (at least IAM, EC2)