REAP-10 Allow setting up per-cluster TLS connections

[rzvoncek]

This PR allows Reaper to use cluster-specific trust/key-stores when using HTTP instead of JMX (meaning Reaper is talking to mgmt-api).

The Reaper's yaml file can now have a new httpManagement.truststoresDir field:

httpManagement:
 enabled: ${REAPER_HTTP_MANAGEMENT_ENABLE}
 keystore: ${REAPER_HTTP_MANAGEMENT_KEYSTORE_PATH}
 truststore: ${REAPER_HTTP_MANAGEMENT_TRUSTSTORE_PATH}
 truststoresDir: ${REAPER_HTTP_MANAGEMENT_TRUSTSTORES_DIR}

This should point to a directory where per-cluster certificates will appear. The certs are expected to have format of clusterName-truststore|keystore.jks.

When adding a cluster, it is now necessary to specify the cluster name via the seed host, so the seed host becomes seedHost@clusterName. The clusterName should match the prefix of in the cert names in truststoresDir. If the cluster name is not provided, Reaper falls back to using the global keystores.

Hot reloading of the cluster-specific stores works the same way as for the global ones.

[github-actions]

No linked issues found. Please add the corresponding issues in the pull request description.
Use GitHub automation to close the issue when a PR is merged

[burmanm]

issue: There's odd logic here. If getTrustStoresDir() is null, then we won't have any RuntimeException.

[burmanm]

Or do you rely on the truststore value to cause the RuntimeException somewhere else?

[rzvoncek]

Yeah I've added it here, in a checkConfiguration(), so that it fails early. Then the stacktrace is smaller and more obvious.

Otherwise, the RuntimeException would happen later, when it's trying to setup the watchers for changes, which in turn happens when actually adding a cluster. The stack traces at this point are incredibly long and nested.

[burmanm]

discussion/issue: I'm not a huge fan of releasing every PR's every commit as image. It really pollutes the Docker Hub. Merging on main is fine, but do we really need these for every PR commit? Especially since this will fail the PR build when a user uses a fork (which would be the default for most users).

If you just want the built image available as part of the PR, then you could use the upload-action so it would be available as download.

[rzvoncek]

I agree that poluting Dcker Hub is no good. I've got my image out and the related k8ssandra-operator PR is passing. I'm ok with dropping this entirely (pushed a commit doing so)

[burmanm]

issue: There's no if here to prevent creating ssl watcher. We would still always create the thread and use Filewatchers, just we wouldn't actually watch any events. If all of these are false, then we shouldn't call createSslWatcher() at all.

[rzvoncek]

Squashed locally, merging.