Use DRPolicy reconciler to pass S3 profiles and secrets to managed clusters

[ShyamsundarR]

Currently the progress on #93 makes it such that an admin has to configure S3 profiles at the hub and on ManagedClusters.

As presented by @vdeenadh the hub can transfer S3 profiles to the ManagedClusters, and this can be done when reconciling DRPolicy as in #79 whenever a new DRPolicy is created.

IOW, Ramen Orchestrator manages the S3 profile configuration of Ramen Manager at the ManagedCluster

[ShyamsundarR]

Need to analyze best possible action here, as DRPC at present does not need the S3 profiles, so change in configuration is possible. Also shifting the profile names to a config map make it non-user friendly for anyone interested in using just the VRG portion of the reconciler.

[hatfieldbrian]

User stories

Drpolicy creation

A hub cluster administrator defines one or more s3 profiles in a RamenConfig ConfigMap each referencing a secret, and a DrPolicy drClusterSet that references, for each of its clusters, one of the s3 profiles. The drpolicy controller arranges for all s3 profiles referenced in a drpolicy's drClusterSet and the secrets they reference to be replicated to each cluster in the drpolicy's drClusterSet.

Drpolicy deletion

A hub cluster administrator deletes a drpolicy. The drpolicy controller gets an exclusive lock and determines if any clusters specified in the drpolicy are specified in any other drpolicies on the cluster, and for each ones that isn't, requests its s3 profiles and the secrets they reference be deleted.

Drpolicy s3 profile reference is modified

A hub cluster administrator changes the reference to an s3 profile in a drpolicy. The drpolicy controller gets an exclusive lock and determines if any drpolicies on the hub that contain a cluster in the modified drpolicy (assuming its cluster names were not modified or the previous ones are known), still reference the old s3 profile or its secret (assuming the old s3 profile name and its secret reference are known). If none do, the s3 profile and its secret are deleted. If at least one drpolicy references either the s3 profile or its secret, the resource is not deleted. The new s3 profile and its secret are deployed to the drpolicy's managed clusters.

Note: drpolicy updates that remove things (e.g. clusters) not referenced by any other drpolicies, currently leak the resources that drpolicy controller deployed because of their presence.

[hatfieldbrian]

This is addressed in pull request #358. Secret configuration data is now tracked by issue #359.

[ShyamsundarR]

Code changes are merged, closing this issue.