K8SSAND-1515 ⁃ Error when parsing the system replication annotation after v1.1.0 upgrade

[adejanovski]

What happened?
After upgrading to v1.1.0, the following error pops up in the operator logs:

1.652681134996166e+09	ERROR	controller.k8ssandracluster	System replication check failed	{"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "dogfood", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/dogfood", "error": "json: cannot unmarshal array into Go value of type int"}
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcile
	/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:133
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
	/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227

The code is trying to unmarshall the initial-system-replication annotation to a map[string]int, but the format we get from v1.0.0 is: k8ssandra.io/initial-system-replication: '{"datacenters":["dc2","dc1"],"replicationFactor":3}'

v1.1.0 rather expects the following format it seems: {'dc1':3, 'dc2':3}

We need to support both formats and migrate the old one to the new one during an upgrade.

Did you expect to see something different?
The upgrade shouldn't trigger any error and the operator should gracefully accept both formats, performing a silent migration to the new annotation format.

How to reproduce it (as minimally and precisely as possible):
Upgrade a cluster from v1.0.x to v1.1.0.

Environment
GKE

• K8ssandra Operator version:

  v1.1.0

  * Kubernetes cluster kind:

  GKE

┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1515
┆priority: Medium