ROX-18431: restore deleted tenant through API

[johannes94]

Description

This PR adds an API endpoint to the admin API of fleetmanager that allows to restore a deleted Central.

The restore operation resets the values of a deleted central in the database that are filled during the provisioning and deletion state machine flow. Then it resets it to provisioning and lets fleetshard-sync recreate the tenant. Additional logic in fleetshard-sync will search for RDS snapshots associated with cluster-id and restore the DB from a final snapshot if it exists.

The restore operation is currently limited, as it only restores the database to a state where the db password and master password needs to be restored. In order to get to a working tenant one has to follow: https://gitlab.cee.redhat.com/stackrox/acs-cloud-service/runbooks/-/blob/3c22117db0652ff52d54179038f625164d352f15/sops/dp-024-lost-tenant-db-password.md#L1

This PR will fix that: #1197

Checklist (Definition of Done)

• Unit and integration tests added
• Added test description under Test manual
• [ ] Documentation added if necessary (i.e. changes to dev setup, test execution, ...) Docs will be updated in SOP repo.
• CI and all relevant tests are passing
• Add the ticket number to the PR title if available, i.e. ROX-12345: ...
• Discussed security and business related topics privately. Will move any security and business related topics that arise to private communication channel.
• Add secret to app-interface Vault or Secrets Manager if necessary
• RDS changes were e2e tested manually
• Check AWS limits are reasonable for changes provisioning new resources

Test manual

Tested on an OSD cluster with all integrations (Route53, AWS RDS Database, RH SSO)

# Populate RH SSO dynamic client secrets, you can get them from app-interface vault of stage
# redhatsso-service.clientId and redhatsso-service.clientSecret files
# Populate Route53 secrets for dev, you can get them from OSCI vault
# aws.route53accesskey $aws.route53secretaccesskey
# Start fleetmanager with external DNS and your cluster config

make binary
make db/teardown db/setup db/migrate

./fleet-manager serve --enable-central-external-certificate --dataplane-cluster-config-file ./dev/config/dataplane-cluster-configuration-infractl-osd.yaml --force-leader --central-idp-client-id ""

# Start fleetshard-sync in another terminal
# Prepare environment
export CLUSTER_NAME=local_cluster                 
export MANAGED_DB_ENABLED
export MANAGED_DB_ENABLED=true
export AWS_AUTH_HELPER=aws-saml
export CREATE_AUTH_PROVIDER=true

# Start fleetshard-sync
./dev/env/scripts/exec_fleetshard_sync.sh

# In another terminal create an central instance
./scripts/create-central.sh

# Wait for central to be provisioned
# With above setup I included all integrations. Route53, RH SSO, RDS so it will take a while
# Once central is up login via Web Browser
# Create an API token in the integration tab of central
# We're going to use this token to ensure DB restore was successful later on so store it in a file
echo $token > token.txt

# Now delete the central tenant and wait for it's deprovisioning flow to complete
export central_id=<central-id>
export OCM_TOKEN=$(ocm token)
./scripts/fmcurl "rhacs/v1/centrals/$central_id?async=true" -XDELETE -v

# While waiting get an admin token
rhoas login --auth-url=https://auth.redhat.com/auth/realms/EmployeeIDP 
export OCM_TOKEN=$(rhoas authtoken)

# Once the deletion went through restore it through admin API
./scripts/fmcurl "rhacs/v1/admin/centrals/$central_id/restore" -XPOST -v

# Monitor restore operation, central will start in preparing and should go to ready. Through FM<>FS interaction.
# Once it's ready call the UI again and Login to test it's working
# Then as a final test use the API token
export ROX_API_TOKEN=(cat token.txt)
export ROX_ENDPOINT=acs-<central-id>.rhacs-dev.com:443
roxctl central whoami

# To run tests locally run:
make db/teardown db/setup db/migrate
make ocm/setup OCM_OFFLINE_TOKEN=<ocm-offline-token> OCM_ENV=development
make verify lint binary test test/integration

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all