ROX-13709: Introduce the external secrets operator on the Data Plane

[kovayur]

Description

This change introduces the external secrets operator (https://external-secrets.io) on the Data Plane clusters.
This allows to load secrets from AWS on the cluster instead of populating them to the deployment script in Github Actions.

The change includes:

1. Deployment script
2. CI / CD pipeline
3. Load fleetshard sync secrets using the ExternalSecret CR.

After we make all the secrets load from the operator we can:

1. Revoke the secret read permission from the Github AWS role
2. Simplify the terraforming chart and its deployment by reducing the number of values.

Checklist (Definition of Done)

• Unit and integration tests added
• Added test description under Test manual
• Documentation added if necessary (i.e. changes to dev setup, test execution, ...)
• CI and all relevant tests are passing
• Add the ticket number to the PR title if available, i.e. ROX-12345: ...
• Discussed security and business related topics privately. Will move any security and business related topics that arise to private communication channel.
• Add secret to app-interface Vault or Secrets Manager if necessary
• RDS changes were e2e tested manually
• Check AWS limits are reasonable for changes provisioning new resources

Test manual

Tested manually on the stage cluster.

[stehessel]

Very nice! Thanks for picking up ROX-13709 in its unrefined state 🚀 . We can do a follow up for other service secrets (like probe and cloudwatch exporter).

What do you think about enabling etcd encryption? That way the secrets are encrypted at rest in the cluster as well as in AWS - see https://docs.openshift.com/container-platform/4.13/security/encrypting-etcd.html on how to do that.

[ebensh]

Sorry for my ignorance here, where do these user IDs come from? I know the default of 1000 but what's the source for these?

[kovayur]

They came from openshift, the default is 1000, but this is outside the allowed range of values. I took the first values from the range

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ebensh, kovayur, porridge, SimonBaeumer, stehessel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [SimonBaeumer,ebensh,kovayur,porridge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kovayur]

What do you think about enabling etcd encryption? That way the secrets are encrypted at rest in the cluster as well as in AWS - see https://docs.openshift.com/container-platform/4.13/security/encrypting-etcd.html on how to do that.

@stehessel raised a follow-up ticket for this: ROX-20619